Download speed from Rigol DS1054Z or similar oscilloscope to a PC - Page 1
Forum section: electronics / testgear

author: RoGeorge
author_rank: Super Contributor
timestamp: November 12, 2022, 07:47:34 am
content: Asking because I've just tried, and it took more than 21 minutes to download all the 24 million data points.The PC can easily download from Internet at 700-800 Mbps, so the bottleneck is not the computer's hardware.- Software: Kubuntu -> VScode (Codium) -> Jupyter Notebooks -> my Python scripts -> python-ivi -> PyVISA -> PyVISA-py -> python-vxi11- Hardware: i7 Desktop -> LAN cable -> 100Mbps router (switch) -> LAN cable -> Rigol DS1104Z oscilloscopeAnybody else tried to download all the data sampled by a Rigol oscilloscope?How fast does that worked for you, and with what software/hardware, please?

author: Fungus
author_rank: Super Contributor
timestamp: November 12, 2022, 09:27:57 am
content: What format did you download it in? CSV?Bottom line: The rigol is an oscilloscope, not a high speed internet device.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 12, 2022, 09:28:56 am
content: Quote from: Karel on November 29, 2016, 04:12:13 pmThat was 8+ years ago. Back then I was experiencing very slow downloads, but only when using Windows. From Linux it was OK. I thought that issue was sorted out. Now, many firmware updates and many OS versions later, I'm seeing about the same problem, but with Linux.Anybody else tried downloading the data samples from a Rigol recently, please?

author: Fungus
author_rank: Super Contributor
timestamp: November 12, 2022, 12:30:40 pm
content: Quote from: Mechatrommer on November 12, 2022, 10:50:33 amWe know nothing at all about how OP is "downloading"the data...

author: Karel
author_rank: Super Contributor
timestamp: November 12, 2022, 02:08:21 pm
content: Using DSRemote it takes exactly 30 seconds (per channel) to download the whole 24Mpts.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 14, 2022, 10:19:03 am
content: Thanks for mentioning DSremote, forgot about that one. Installed DSremote in Kubuntu 22.04 LTS, and it takes only 30 seconds to download data in its Wave Inspector.At first I was trying from Python-IVI, which is a layer on top on PyVISA, as described in the OP. It was way too slow, so this weekend tried sending SCPI commands from C, by calling into the 'liblxi.so' library (liblxi is available in most of the official repos, "sudo apt install liblxi-dev" or from sources athttps://github.com/lxi-tools/liblxi). Then tried with DSremote, and DSremote is still 3 times faster than anything else I've tried.Always used RAW mode and BYTES, so 24MB of data to be downloaded for all the 24MSa of ADC data. No matter the size of each consecutive download batches, the duration benchmark is for a complete 24Msa download.23 minutes - Python scripts, IVI calls -> python-ivi -> PyVISA -> PyVISA-py -> python-vxi11 -> LAN -> DS1104Z134 seconds - C program, SCPI requesting batches of 125'000 bytes at once -> lib lxi -> lib tirpc -> LAN -> DS1104Z86 seconds - C program, SCPI requesting batches of 250'000 bytes at once (maximum size according to the DS1000Z programming guide) -> lib lxi -> lib tirpc -> LAN -> DS1104Z65 seconds - C program, SCPI requesting batches of 750'000 bytes at once -> lib lxi -> lib tirpc -> LAN -> DS1104Z55 seconds - C program, SCPI requesting batches of 1'000'000 bytes at once -> lib lxi -> lib tirpc -> LAN -> DS1104Z30 seconds :o - DSremote downloading all the samples to be saved or viewed in its waveform inspectorIt's interesting how DSremote is at least twice than any other setup. I've looked at the LAN packets with Wireshark, and it seems DSremote is not using VXI11 protocol for download, it is using plain TCP on the port 5555. The SCPI commands from DSremote are as expected:Code:STOP:*OPC?WAV:SOUR CHAN1*OPC?WAV:FORM BYTE*OPC?WAV:MODE:RAW*OPC?WAV:YINC?WAV:YREF?WAV>YOR?WAV:STAR 1WAV:OPC?WAV:STOP 250000WAV:OPC?WAV:DATA?// incoming first 250'000 ADC samplesWAV:STAR 250001\0x0a*OPC?\0x0a// (expected response from *OPC is 1\0x0a)WAV:STOP 500000\0x0a*OPC?\0x0aWAV:DATA?\0x0a// (data response is #9000250000<250kB_of_data_over_many_TCP_packets_of_mostly_65535_TCP_bytes_each>\0x0a// and so on until all the wanted 24 million samples are fetched// ...- SCPI terminator is always the 0x0a char <LF> (beware printf("\n") in Windows is two cars, <CR><LF>, printf("%c", 10) might be safer)- OPC? (operation completed) is probed by DSremote after each command (which I didn't probe, maybe I should)- timeout in the DSremote settings was left as found, 50ms, but if I put the same timeout in my C code, the value is too small and leads to timeout errors sooner or later. I had to use 2000ms instead of 50ms.This is my fastest so far, with C and liblxi, which is still about 3 times slower than DSremote:Code:// liblxi git at https://github.com/lxi-tools/liblxi// to install it from the ubuntu repository (lxilib-dev is in most of the Linux distros repos and in FreeBSD):// sudo apt install liblxi-dev// headers in /usr/include/lxi.h// .so lib in /usr/lib/x86_64-linux-gnu/liblxi.so// display all installed files with 'dpkg -L liblxi-dev' or// sudo apt install apt-file// sudo apt-file update// (sudo???) apt-file list liblxi-dev// no need to append LIBRRY_PATH or LD_LIBRARY_PATH// man lxi_<TAB><TAB>// lxi_connect lxi_discover lxi_init lxi_send lxi_disconnect lxi_discover_if lxi_receive// compile and run this 'demo.c' from a Linux terminal (!!! -l options _must_ be positioned after the .c sources, not before)// gcc -Werror -Wfatal-errors ./demo.c -llxi -ltirpc && time ./a.out#include <stdio.h>#include <stdlib.h>#include <string.h>#include <time.h>#include <lxi.h>int main(){// 12 bytes preamble + max of 24 mil bytes of ADC samples + 1 byte for the end of string terminator#define CHARS_BUFFER_MAX 24000000L // in specs max Rx in a single chunk is 250k chars for Rigol DS1054Zchar *rx24MB = NULL; // declare a pointer to char, and initialize it with NULLrx24MB = malloc(CHARS_BUFFER_MAX * sizeof *rx24MB); // allocated memory on the heap for rx24MBif (!rx24MB) { // !!! always check the memory was allocated successfully !!!fputs ("ERROR: allocation memory failed for the 'rx24MB' buffer, exiting.", stderr);exit (EXIT_FAILURE);}int device, length, timeout = 2000; // timeout is in mschar response; // allocated on the stack, segmentation fault if too large, Linux max stack is 8MiBchar *command = "*IDN?";lxi_init(); // Initialize LXI librarydevice = lxi_connect("192.168.1.3", 0, "inst0", timeout, VXI11); // Connect to LXI devicelxi_send(device, command, strlen(command), timeout); // Send SCPI command ("*IDN?")lxi_receive(device, response, sizeof(response), timeout); // Rx until "\n", or sizeoff(response) chars, or timeoutprintf("%s\n", response);// exit if no DS1054Z// exit(0);// enable only ch1command = ":CHAN1:DISP 1";lxi_send(device, command, strlen(command), timeout);command = ":CHAN2:DISP 0";lxi_send(device, command, strlen(command), timeout);command = ":CHAN3:DISP 0";lxi_send(device, command, strlen(command), timeout);command = ":CHAN4:DISP 0";lxi_send(device, command, strlen(command), timeout);// ??? here to insert ":RUN""// set acquisition mode to max 24_000_000 points memory depth// "auto", 12K, 120K, 1.2M, 12M, 24M"long mdep = 24000000L;command = ":RUN";lxi_send(device, command, strlen(command), timeout);// sprintf(command, ":ACQ:MDEP %li", mdep);// sprintf(command, ":ACQ:MDEP %li%s", mdep, "\0");command = ":ACQ:MDEP 24000000";lxi_send(device, command, strlen(command), timeout);// wait for triggereddo {command = ":TRIGger:STATus?";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);} while (response != 'T');// then wait for ADC to aquire enough dataclock_t stop_time;printf("wait %ld\n", clock());stop_time = clock() + 30000; //30 secondsprintf("beep %ld\n", stop_time);while (clock() < stop_time) ;printf("go %ld\n", clock());// go to stop mode and prepare to dl 24MSacommand = ":STOP";lxi_send(device, command, strlen(command), timeout);command = ":WAV:SOUR CHAN1";lxi_send(device, command, strlen(command), timeout);command = ":WAV:MODE RAW";lxi_send(device, command, strlen(command), timeout);command = ":WAV:FORM BYTE";lxi_send(device, command, strlen(command), timeout);// char c2 = {0};char c2;// loop to dl all samples in chunks of (max) 250_000 samples// set start-stop indexes for next chunklong rx_len, chunk_len;// long chunk_start, chunk_stop, chunk_size = 125000L; //dl 24Msa/134slong chunk_start, chunk_stop, chunk_size = 250000L; //dl 24Msa/86s max from specs// long chunk_start, chunk_stop, chunk_size = 750000L; //dl 24Msa/65s// long chunk_start, chunk_stop, chunk_size = 1000000L; //dl 24Msa/56s preferred ???// long chunk_start, chunk_stop, chunk_size = 1175000L; //dl 24Msa/54slong samples_to_dl = 24000000L;if (samples_to_dl > mdep) {samples_to_dl = mdep;}for(chunk_start = 1; chunk_start < samples_to_dl; chunk_start += chunk_size) {sprintf(c2, ":WAV:STAR %li", chunk_start);puts(c2);rx_len = lxi_send(device, c2, strlen(c2), timeout);printf("ack_len=%ld\n", rx_len);//at the last chunk, chunk_len might be shorter than chunk_sizeif (chunk_start + chunk_size > samples_to_dl) {chunk_len = samples_to_dl - chunk_start + 1;chunk_stop = samples_to_dl;} else {chunk_len = chunk_size;chunk_stop = chunk_start + chunk_size - 1;}sprintf(c2, ":WAV:STOP %li", chunk_stop);puts(c2);rx_len = lxi_send(device, c2, strlen(c2), timeout);printf("ack_len=%ld\n", rx_len);command = ":WAV:DATA?";puts(command);rx_len = lxi_send(device, command, strlen(command), timeout);printf("ack_len=%ld\n", rx_len);rx_len = lxi_receive(device, (char *)(&rx24MB[chunk_start-1]), chunk_len, timeout);printf("rx_data_bytes=%ld\n", rx_len);int err = 0;if(rx_len != chunk_len) {printf("ERROR - received too few bytes: ");err = 1;}printf("chunk_len=%li rx_len=%li\n", chunk_len, rx_len);printf("\n");if(err) exit(err);}// Disconnectlxi_disconnect(device);}Not sure if I am doing something wrong, or if I should just use TCP instead of VXI11?

author: Fungus
author_rank: Super Contributor
timestamp: November 14, 2022, 10:43:50 am
content: Quote from: RoGeorge on November 14, 2022, 10:19:03 amYes.

author: alm
author_rank: Super Contributor
timestamp: November 14, 2022, 12:37:58 pm
content: Quote from: RoGeorge on November 14, 2022, 10:19:03 amPython-IVI is transport-agnostic, so I'd think that you can tell the IVI driver to connect using TCP instead of VXI-11 using a differentVISA resource name. I don't see why you'd need to use C for a throughput of maybe 2 MB/s.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 14, 2022, 01:22:07 pm
content: I used C because the SW stack from python was heaving too many layers and I didn't know which one was slowing down the transfers. The simplest way I was aware of (other than making direct socket calls) was to use liblxi from C.I did met some similar issues (10-100 slower transfers than it should be) about 7-8 years ago, when I was using SCPI sent to a TCP socket, directly from Python with no drivers, or IVI/VISA/VXI or alike. Back then it was happening in Windows only, because of some specific algorithm Windows was using when it was increasing the TCP packets size. That was broken by design for Windows, but it was working fine from Linux.Now I am seeing those 20+ minutes long transfers in Linux. Don't know yet if it's caused by the same issue from 8 years ago, or if it's something different. Maybe the wrong by designed RWIN (TCP Receive WINdow) algorithm used by Windows back then was meanwhile added as a new Linux "feature", IDK.https://hackaday.io/project/5807-driverless-rigol-ds1054z-screen-capture-over-lan/log/18807-solved-a-painfully-slow-lan-on-windows-and-a-wasted-weekend

author: alm
author_rank: Super Contributor
timestamp: November 14, 2022, 01:53:32 pm
content: Quote from: RoGeorge on November 14, 2022, 01:22:07 pmIs that over VXI-11 or over TCP sockets (TCPIP0::host address::port::SOCKET)? VXI-11 is probably not optimized for high throughput, but plain TCP sockets should be pretty straight-forward.

author: DiTBho
author_rank: Super Contributor
timestamp: November 14, 2022, 04:21:23 pm
content: too confusing

author: DiTBho
author_rank: Super Contributor
timestamp: November 14, 2022, 04:42:40 pm
content: I don't have a Rigol DS1054Z, butsigrokseems to have support, seehere

author: RoGeorge
author_rank: Super Contributor
timestamp: November 14, 2022, 06:40:05 pm
content: Quote from: alm on November 14, 2022, 01:53:32 pmWireshark was telling the LAN packets were of type VXI-11 protocol, most probably the slow transfer was for VXI. In fact, it might be the only one supported, I don't know how to configure IVI to use SOCKET instead of VXI-11.I've just tried "TCPIP0::host address::port::SOCKET" and it breaks with the err message "unrecognized resource string format". By looking inside the code, the regular expressions used to validate the resource string doesn't know about the word "SOCKET", can only end in "INSTR". Same in the man page, all examples are of type "TCPIP................INSTR"https://www.systutorials.com/docs/linux/man/1-pythoniviIt is still possible to use python-ivi for all the other commands, and write a C function to fetch the ADC data points only using SOCKET and forcing the TCP packets size, the oscilloscope allows multiple connections. Or I can dig in my own python2 repos from 8 years ago and check if the WAVE:DATA? scripts from back then are still fast (in Linux).IVI is very appealing, but for some reason it didn't get enough traction, and now it feels like abandonware, the last accepted pull in the main project was in 2016 or so. For example, two years ago I had to merge from many forks and found no python-ivi driver for my signal generator. Now, I had to search again through 108 forks and merge the most active (active in the sense that they appear to have extra python-ivi drivers), thought there is still no driver for my generator, but there is a generator driver from another instrument, and it happens that the main controls are the same, so at least I can change the frequency from python-ivi and do the rest from buttons.I really like python-ivi, but it sunk me way too much time. After all, I don't think any body else will reuse my automated test software, not even myself, so I can just use SCPI. It's been 10 days since I've started to fool around with python-ivi and still tinkering around it instead of working at the data processing, which was the actual goal of the project.

author: alm
author_rank: Super Contributor
timestamp: November 14, 2022, 07:16:45 pm
content: Quote from: RoGeorge on November 14, 2022, 06:40:05 pmReading thesource code, this regular expression is for the resources python-ivi can handle directly without PyVisa, like VXI-11. Anything else should be passed to PyVisa, unless PyVisa is not imported, which sounds like is the case for you (the exception on line 1837).Quote from: RoGeorge on November 14, 2022, 06:40:05 pmFor me python-ivi is still worth it because it provides a fairly sensible way to abstract instruments, so I can use the same code for a Tektronix scope and a Lecroy scope, for example, which as a hobbyist with a heterogeneous lab with instruments from different manufacturers is very helpful. Even though I've had to develop most of the drivers that I use myself, this was a one-time effort. I don't just have three identical instruments to throw at the same problem, so if I need multiple scopes, power supplies or multimeters at the same time, they'll probably of different brands and vintages. I have not found an abstraction I liked better than python-ivi yet, and I certainly don't feel like inventing my own.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 16, 2022, 01:39:58 pm
content: It looks like either python-ivi or pyvisa will use the same python-vxi11 module (import vxi11) to connect to an instrument over LAN. I've tested and they are both very slow, it takes 23minutesto fetch 24MSa.Code:# pip install python-vxi11# Successfully installed python-vxi11-0.9import vxi11dso = vxi11.Instrument('192.168.1.3')# alternatives (transfer speed is the same):# dso = vxi11.Instrument("TCPIP::192.168.1.3::INSTR")print(dso.ask('*IDN?'))dso.write(':RUN')while dso.ask('*OPC?') != '1': print('Waiting for command to finish')dso.write(':ACQ:MDEP 24000000') # !!! can only be set when in :RUN modewhile dso.ask('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:MODE MAX') # !!! 1200 points in :RUN mode, all MDEP points in STOP modewhile dso.ask('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:FORM BYTE')while dso.ask('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:SOUR CHAN1')while dso.ask('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:STAR 1')while dso.ask('*OPC?') != '1': print('Waiting for command to finish')dso.write(':STOP')while dso.ask('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:STOP 250000') # !!! will be ignored and always 1200 in :RUN modewhile dso.ask('*OPC?') != '1': print('Waiting for command to finish')wvfm_pre = dso.ask('WAV:PRE?')while dso.ask('*OPC?') != '1': print('Waiting for command to finish')print(wvfm_pre)print(dso.ask('WAV:MODE?'), dso.ask('WAV:STAR?'), dso.ask('WAV:STOP?'))dso.write(':STOP')while dso.ask('*OPC?') != '1': print('Waiting for command to finish')import timet0 = time.time()wvfm = dso.ask_raw(b':WAV:DATA?') # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP modet1 = time.time() - t0print('Time to fetch the first waveform chunk: ' , t1)print(type(wvfm), len(wvfm))while dso.ask('*OPC?') != '1': print('Waiting for command to finish')to_fetch = 1_000_000 # fetch only 1Msa of ADC samplesMAX_CHUNK = 250_000M_DEPTH = int(dso.ask(':ACQ:MDEP?'))all_wvfm = bytes()t0 = time.time()for i in range(0, min(to_fetch, M_DEPTH), MAX_CHUNK):dso.write(':WAV:STAR ' + str(i+1))while dso.ask('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:STOP ' + str(i + min(MAX_CHUNK, M_DEPTH, to_fetch-i)))while dso.ask('*OPC?') != '1': print('Waiting for command to finish')wvfm = dso.ask_raw(b':WAV:DATA?') # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP modeall_wvfm += wvfm[12:]print(len(wvfm))t1 = time.time() - t0print('Time to fetch ' + str(to_fetch) + ' ADC samples using "python-vxi11": ' + str(t1))dso.close()If I open a connection with SOCKET, or if I use something like the 'socketscpi' module, it fetches 24MSa in 23 seconds.Code:# socketscpi is using direct TCP socket instead of VXI-11# pip install socketscpi# Successfully installed socketscpi-2022.8.0# might work as a backend for PyVISA ????? for LXI instruments# https://github.com/morgan-at-keysight/socketscpi/blob/master/examples.pyimport socketscpidso = socketscpi.SocketInstrument(ipAddress='192.168.1.3', port=5555, timeout=1)# dso = socketscpi.SocketInstrument(ipAddress='192.168.1.3', port=5555, globalErrCheck=True)print(dso.instId)dso.write(':RUN')while dso.query('*OPC?') != '1': print('Waiting for command to finish')dso.write(':ACQ:MDEP 24000000') # !!! can only be set when in :RUN modewhile dso.query('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:MODE MAX') # !!! 1200 points in :RUN mode, all MDEP points in STOP modewhile dso.query('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:FORM BYTE')while dso.query('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:SOUR CHAN1')while dso.query('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:STAR 1')while dso.query('*OPC?') != '1': print('Waiting for command to finish')dso.write(':STOP')while dso.query('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:STOP 250000') # !!! will be ignored and always 1200 in :RUN modewhile dso.query('*OPC?') != '1': print('Waiting for command to finish')wvfm_pre = dso.query('WAV:PRE?')while dso.query('*OPC?') != '1': print('Waiting for command to finish')print(wvfm_pre)print(dso.query('WAV:MODE?'), dso.query('WAV:STAR?'), dso.query('WAV:STOP?'))dso.write(':STOP')while dso.query('*OPC?') != '1': print('Waiting for command to finish')import timet0 = time.time()wvfm = dso.query(':WAV:DATA?') # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP modet1 = time.time() - t0print('Time to fetch the first waveform chunk: ' , t1)print(type(wvfm), len(wvfm))while dso.query('*OPC?') != '1': print('Waiting for command to finish')to_fetch = 1_000_000 # fetch only 1Msa of ADC samplesMAX_CHUNK = 250_000M_DEPTH = int(dso.query(':ACQ:MDEP?'))all_wvfm = b''t0 = time.time()for i in range(0, min(to_fetch, M_DEPTH), MAX_CHUNK):dso.write(':WAV:STAR ' + str(i+1))while dso.query('*OPC?') != '1': print('Waiting for command to finish')dso.write(':WAV:STOP ' + str(i + min(MAX_CHUNK, M_DEPTH, to_fetch-i)))while dso.query('*OPC?') != '1': print('Waiting for command to finish')wvfm = dso.query(':WAV:DATA?') # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP modeb = bytes(wvfm[12:], 'latin_1')all_wvfm += bprint(len(wvfm))t1 = time.time() - t0print('Time to fetch ' + str(to_fetch) + ' ADC samples using "socketscpi": ' + str(t1))dso.close()The downside is if I send a wrong command I have to power cycle the oscilloscope.My oscilloscope is Rigol DS1104Z (in python-ivi its driver module is 'rigol.rigolDS1054'), it has the fixed IP 192.168.1.3, and the port for SOCKET access is 5555.Do you have a working code example, please, for connecting to the oscilloscope from python-ivi, or even from PyVISA, but using SOCKET instead of VXI11, and then print the result of the '*IDN?'

author: alm
author_rank: Super Contributor
timestamp: November 16, 2022, 02:25:53 pm
content: I don't have any Rigol scope to test on, so I used netcat to create a listening TCP socket:nc -l -p 1052. Then I installed python-ivi fromyour Github, removed all the Python-2 style print statements that would give errors in Python 3 (afterfrom __future__ import print_function,print(...)with parentheses should work in any Python version released after 2008), installed pyvisa and pyvisa-py, and made a small change toivi/interfaces/pyvisa.pylines 32-39 and replaced the wordvisabypyvisa. Apparently the module changed names in the past six years. And then it seems to work fine for me:Code:% python>>> import ivi>>> rig = ivi.rigol.rigolDS1054Z('TCPIP0::localhost::1052::SOCKET')# This was necessary because I'm using a Linux terminal and not a real DS1054 scope>>> rig._interface.instrument.read_termination = '\n'>>> rig._interface.instrument.write_termination = '\n'>>> rig.identity.instrument_model'DS1102E'In netcat I'd see:% nc -l -p 1052*CLSRIGOL TECHNOLOGIES,DS1102E,DS1EB104702974,00.02.01.01.00*IDN?Red is what the driver sent, and green is what I typed to simulate the response of a scope.*CLSis what the DS1054Z driver sends when you instantiate it, andRIGOL TECHNOLOGIES,...is what I typed as a pre-emptive response to the*IDN?query since I can't type fast enough to stay within the timeout. I'm pretty sure that the same Python code, probably without changing the termination, would work with a real scope.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 16, 2022, 05:09:43 pm
content: Sorry for the invalid Python2 prints, not mine, I've removed them only locally, now removed them from my git fork, too.The problem is that I get invalid resource string for something like:>>> rig = ivi.rigol.rigolDS1054Z('TCPIP0::localhost::1052::SOCKET')Code:...python-ivi_RoGeorge_fork/python-ivi/ivi/ivi.py", line 1817, in _initializeraise IOException('Invalid resource string')ivi.ivi.IOException: Invalid resource string>>>If that resource string worked for you, then you must be having one of the proprietary VISA libraries installed, something like NI-VISA, Keysight VISA, R&S VISA, Tektronix VISA, and python-IVI or maybe pyvisa detects the closed source library and sends the resource string to it.I don't have any of the proprietary VISA libraries, only the FOSS python modules: python-ivi, python-vxi11, pyvisa and pyvisa-py. IIRC pyvisa also has some options to configure its backend.Also I've noticed your prompt is % (% prompt is usually for csh). What OS did you used to test the line "ivi.rigol.rigolDS1054Z('TCPIP0::localhost::1052::SOCKET')"?TL DR;'TCPIP0::localhost::1052::SOCKET' doesn't work on my Kubuntu 22.04 LTS, I have no proprietary NI-VISA or alike installed. Any hints what am I missing, or mis-configured, or what should I install to make my python-IVI work with SOKET connection just like yours, please?

author: alm
author_rank: Super Contributor
timestamp: November 16, 2022, 05:35:47 pm
content: This was on Ubuntu 22.04.1 LTS without any proprietary VISA installed, just PyVISA and PyVISA-py. These are the only modules installed inside the virtual environment where I ran these commands:Code:python = "^3.10" (resolved to Python 3.10.6)PyVISA = "^1.12.0" (resolved to 1.12.0)PyVISA-py = "^0.5.3" (resolved to 0.5.3)numpy = "^1.23.4" (resolved to 1.23.4)python-ivi = {git = "https://github.com/RoGeorge/python-ivi.git"}and as indirect dependency of PyVISA typing-extensions 4.4.0PyVISA used the PyVISA-py backend without any configuration on my side. This is with using PyVISA directly:Code:>>> import pyvisa>>> rm = pyvisa.ResourceManager()>>> res = rm.open_resource('TCPIP0::localhost::1052::SOCKET')>>> res.read_termination = '\n'>>> res.write_termination = '\n'>>> res.write('*IDN?')7>>> res.read()'ME.' (this is just what I typed in netcat)If this doesn't work for you, then maybe try explicitly setting the PyVISA-py backend as describedhere. I did have to modifyivi/interface/pyvisa.pyas I described above before python-ivi would detect pyvisa. Without that modification I also got theInvalid resource stringerror.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 16, 2022, 08:31:39 pm
content: Thanks for pointing to the visa -> pyvisa name changes, wouldn't have noticed that too soon, if ever!Now it can open a SOCKET connection from python-ivi.

author: alm
author_rank: Super Contributor
timestamp: November 16, 2022, 08:46:02 pm
content: Yeah, I guess the module was called "visa" back in 2015 when that code was written. I've never run into it myself because all devices I've controlled so far have a GPIB interface that I access via VXI-11 using a LAN to GPIB bridge (HP/Agilent, Tektronix, etc). So I have no use for anything but VXI-11 for now.Nice, does it work with the DS1054Z now?

author: RoGeorge
author_rank: Super Contributor
timestamp: November 17, 2022, 04:15:37 pm
content: The connection, yes it works, the PythonIVI driver for DS1054Z, not really. For example I was expecting to see the <LF> (0x0a) at the end of the SCPI commands, but the PC doesn't send 0x0a in the LAN packets. The oscilloscope replies with the expected 0x0a terminator. Will look closer these days.

author: alm
author_rank: Super Contributor
timestamp: November 17, 2022, 04:21:50 pm
content: Did you try playing with read and write termination attributes that I set in my samples? I'd expect setting write termination to '\n' to fix the problem of missing line feeds when sending to the scope.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 17, 2022, 04:47:58 pm
content: Only took a brief look, it was very late yesterday. The read or write_termination are from pyvisa, ivi doesn't seem to have them. I guess I should configure the pyvisa backend before making a connection from ivi, not sure how to do it.There is not much documentation, and I'm not a programmer. Digging through code is much more difficult then reading a manual. Overriding or extending classes bamboozles me a lot, since I don't have yet a clean map of the objects used in PythonIVI.

author: alm
author_rank: Super Contributor
timestamp: November 17, 2022, 05:51:23 pm
content: See my earlier posts on how to set termination through python-ivi.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 17, 2022, 11:14:15 pm
content: I don't know how to apply read/write_termination before opening the instrument.If I open the instrument first, by default the driver sends '*CLS' without the 0x0a terminator, which hangs the oscilloscope communication, and can recover only by a power cycle. By looking through sources, found out that there are optional parameters possible to send when opening a new connection, one of the parameters being to not emit a "*CLS" command by default.Code:import iviivi.set_prefer_pyvisa()dso = ivi.rigol.rigolDS1104Z(resource='TCPIP0::192.168.1.3::5555::SOCKET', id_query=False, reset=False)dso._interface.instrument.read_termination = '\n'dso._interface.instrument.write_termination = '\n'dso.identity.instrument_modeldso._ask('*IDN?\n')while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')dso._write(':ACQ:MDEP 24000000\n') # !!! can only be set when in :RUN modewhile dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')dso._write(':WAV:MODE MAX\n') # !!! 1200 points in :RUN mode, all MDEP points in STOP modewhile dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')dso._write(':WAV:FORM BYTE\n')while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')dso._write(':RUN\n')dso._write(':WAV:SOUR CHAN1\n')while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')dso._write(':WAV:STAR 1\n')while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')dso._write(':STOP\n')while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')dso._write(':WAV:STOP 250000\n') # !!! will be ignored and always 1200 in :RUN modewhile dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')wvfm_pre = dso._ask('WAV:PRE?\n')while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')print(wvfm_pre)print(dso._ask('WAV:MODE?\n'), dso._ask('WAV:STAR?\n'), dso._ask('WAV:STOP?\n'))dso._write(':STOP\n')while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')import timet0 = time.time()wvfm = dso._ask_raw(b':WAV:DATA?\n') # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP modet1 = time.time() - t0print('Time to fetch the first waveform chunk: ' , t1)print(type(wvfm), len(wvfm))while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')to_fetch = 24_000_000 # fetch only 1Msa of ADC samplesMAX_CHUNK = 250_000M_DEPTH = int(dso._ask(':ACQ:MDEP?\n'))all_wvfm = b''t0 = time.time()for i in range(0, min(to_fetch, M_DEPTH), MAX_CHUNK):dso._write(':WAV:STAR ' + str(i+1) + '\n')while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')dso._write(':WAV:STOP ' + str(i + min(MAX_CHUNK, M_DEPTH, to_fetch-i)) + '\n')while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')wvfm = dso._ask_raw(b':WAV:DATA?\n') # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode# b = bytes(wvfm[12:], 'latin_1')# all_wvfm += ball_wvfm += wvfm[12:]print(len(wvfm))t1 = time.time() - t0print('Time to fetch ' + str(to_fetch) + ' ADC samples using "socketscpi": ' + str(t1))dso.close()This makes possible to connect to an instrument without sending any (wrong terminated) commands, then configure read/write_termination, which worked. However, the read/write_termination settings seems to apply only for ivi functions. For example, if I use osc._ask_raw(':WAV:DATA?\n'), then I must add the 0x0a manually, as '\n'.I had to look more into the ivi sources. Maybe there is a way to set the automatic addition of the 0x0a terminator in pyvisa-py, so I won't end with a mixture of functions where some requires 0x0a while others don't.The transfer speed is 65 seconds to fetch 24MSa.Not bad, but still 3 times slower than the 22 seconds I get when downloading withhttps://github.com/morgan-at-keysight/socketscpiI wonder if it would be possible to use 'socketscpi' instead of 'pyvisa', or maibe to configure 'pyvisa' to use 'socketscpi' instead of 'pyvisa-py'.I've uploaded my current PythonIVI changes on gitlab,https://gitlab.com/RoGeorge/python-ivi, because github got crazy and doesn't let me any more to remote update my own repositories, unless I got a security token from them, which I don't want. Adios GitHub, hello GitLab.

author: alm
author_rank: Super Contributor
timestamp: November 17, 2022, 11:37:30 pm
content: Quote from: RoGeorge on November 17, 2022, 11:14:15 pmMaybe callingdso._write('\n')right after instantiating the DS1104Z object would remove the hang? This would send the termination character the scope is waiting for.Quote from: RoGeorge on November 17, 2022, 11:14:15 pmLooking at the PyVISA source,writeadds a termination character, andwrite_rawnot. So it's not a matter of ivi vs non-ivi functions, but ofwrite_raw()not adding termination, because you're saying you want to write verbatim what you send. Does it work if you usedso._write(':WAV:DATA?')and thenoutput = dso._read_raw()? ask is nothing more than write + read.Quote from: RoGeorge on November 17, 2022, 11:14:15 pmSettingdso._interface.instrument.write_terminationalready sets the terminator in PyVISA, not ivi.dso._interface.instrumentis the PyVISA resource.Quote from: RoGeorge on November 17, 2022, 11:14:15 pmChanging Python-ivi to use socketscpi would probably be a big change. What you could consider is just like python-ivi currently supports using python-vxi11 directly without going through PyVISA, you could also add support for calling the python socket library directly for ::SOCKET resources without going through PyVISA. This would involve writing an instrument class around the socket library likethe PySerial interface, and some minor changes to ivi.py aroundhereto add a case where it instantiates the newly created PySocket instrument instead of the vxi11 of PyVISA instruments.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 18, 2022, 09:08:24 am
content: Talking about configuring PyVISA to use 'socketscpi' as its LAN back-end because of this recent commit into socketscpi, commit that changed the old names of the socketscpi functions in order to match the pyvisa syntax,https://github.com/morgan-at-keysight/socketscpi/commit/3d8bea9b012c03a29d6676b60a21ac7e5a655ae4.By the commit description, 'socketscpi' might be already usable as a backend for PyVISA, just like now it is possible to force PyVISA to use 'pyvisa-py' instead of proprietary ni-visa shared libs (when both ni-visa and pyvisa-py are installed) like in this example fromhttps://pyvisa.readthedocs.io/projects/pyvisa-py/en/latest/index.htmlQuoteCode:# pyvisa usage exampleimport pyvisa as visa# rm = visa.ResourceManager() # will look for proprietary ni-visa or alike first, if installedrm = visa.ResourceManager('@py') # will ignore any installed VISA libs, and use pyvisa-py insteadinst = rm.open_resource('<VISA_format_resource_of_an_instrument_to_open_here>')print(inst.query("*IDN?"))I don't know how to tell PyVISA to use socketscpi instead of pyvisa-py, or if socketscpi is ready to use as a pyvisa backend or not. Will try to learn that by digging through sources. Does this look feasible as a fix (python-ivi with prefer pyvisa --> pyvisa with forced backend socketscpi --> socketscpi using python's SOCKET)?

author: RoGeorge
author_rank: Super Contributor
timestamp: November 18, 2022, 01:34:32 pm
content: Got 24MSamples in 22 seconds, with python-ivi, pyvisa, pyvisa-py and SOCKET connection!!!Found what was making 'pyvisa-py' 3 times slower than 'socketscpi' by looking side by side at the opening socket parameters in 'pyvisa-py' vs 'socketscpi'. It was the default value of the TCP_NODELAY. The download becomes fast if I add this extra lineCode:self._set_tcpip_nodelay(constants.VI_ATTR_TCPIP_NODELAY, True)at the end of the '_connect()' definition, inside the file lib/python3.10/site-packages/pyvisa_py/tcpip.py. It's the line number five hundred and something in the modules installed with pip, probably would have to be inserted after line 840 if the same change were to be added in the github source file:https://github.com/pyvisa/pyvisa-py/blob/main/pyvisa_py/tcpip.py#L840However, I suspect there might be a way to set the nodelay parameter as True, but I don't know enough Python and/or OOP to figure myself the syntax for setting TCPIP NODELAY from the end user code, without changing the sources inside pyvisa-py files.Any idea how to set the nodelay parameter using python-ivi?

author: alm
author_rank: Super Contributor
timestamp: November 18, 2022, 09:23:53 pm
content: Quote from: RoGeorge on November 18, 2022, 09:08:24 amI know that this might be a red herring at this point, but when I said earlier that I thought integrating socketscpi with PyVISA, I was confusing socketscpi with another library, so my statement that it would be hard was wrong. It does not look super hard, but would require looking at how PyVISA-py registers itself with PyVISA, and doing the same with a class that wraps socketscpi.Quote from: RoGeorge on November 18, 2022, 01:34:32 pmCongratulations on finding this! It makes sense that it's a socket option that has such a performance impact. It suggests me that there's something sub-optimal in the Rigol scope about assembling the data into TCP packets, but this is a good workaround.Quote from: RoGeorge on November 18, 2022, 01:34:32 pmIt took a bit of digging, but I think this should work:dso._interface.instrument.visalib.sessions[dso._interface.instrument.session]._set_tcpip_nodelay(...)

author: RoGeorge
author_rank: Super Contributor
timestamp: November 19, 2022, 12:04:57 am
content: Quote from: alm on November 18, 2022, 09:23:53 pmWow, thanks, I've tried today a thousand combinations and couldn't find any that works. Now it gets all the 24 million samples in 22 seconds, connected like this:Code:import ivi# 24Msa in 22 secondsivi.set_prefer_pyvisa()dso = ivi.rigol.rigolDS1104Z(resource='TCPIP0::192.168.1.3::5555::SOCKET', id_query=False, reset=False)from pyvisa import constantsdso._interface.instrument.visalib.sessions[dso._interface.instrument.session]._set_tcpip_nodelay(constants.VI_ATTR_TCPIP_NODELAY, True)# 24Msa in 16 minutes# dso = ivi.rigol.rigolDS1104Z(resource='TCPIP0::192.168.1.3::INSTR', id_query=False, reset=False)Don't celebrate yet, (re)found one more issue!- A (telnet) data transfer will drop if inside any data packets it receives a 0x00. This is some RFC spec (for text Telnet IIRC).- The ADC samples are coming as bytes, which means the incoming data packets will be truncated if one of the bytes is zero, because in the Telnet RFC 0x00 is considered a terminator, and will drop any bytes coming after a 0x00.- This issue is only observed when the input signal in the ADC is less than 5 divisions on the screen (ADC outputs 0x7f for zero volts, and 0x00 for any input voltage that is -5*volts/div or under)- Both the pyvisa->pyvisa-py->socket and the socketscpi are affectedDon't know how to overcome this, or if it is possible at all to open some other kind of socket connection, one that doesn't consider 0x00 as a terminator.From when I've written my own SCPI scripts, I had to modify the Python's telnet module such that it won't end a data transfer at the first 0x00 in a data stream. Since the 0x00 as a terminator is part of the RFC854 spec, this modification can not be merged upstream, so I've just made a local copy of the telnet.py (module) file, and modified the local copy to disregard 0x00 as terminator (commented out those two "if c==" lines:https://github.com/RoGeorge/DS1054Z_screen_capture/blob/master/telnetlib_receive_all.pyCode:try:while self.rawq:c = self.rawq_getchar()if not self.iacseq:#if c == theNULL:# continue#if c == "\021":# continueif c != IAC:buf[self.sb] = buf[self.sb] + ccontinueelse:self.iacseq += celif len(self.iacseq) == 1:Not elegant, but it worked. The plan was to remove the telnetlib.py module later, and use a SOCKET connection instead of a Telnet one, though never implemented that/TL;DRSo far PythonIVI using SOCKET and NO_DELAY is the fastest one, but it only works if the oscilloscope trace is nicely contained inside the screen.Do you happen to know any SOCKET parameter, so to transfer a specified number of bytes until all bytes were transferred (or timeout has occurred), but for binary, such that 0x00 is not considered a data terminator?

author: lundmar
author_rank: Frequent Contributor
timestamp: November 19, 2022, 02:41:58 am
content: RoGeorge,FYI - liblxi also supports connecting to instruments using raw sockets. Simply use 'RAW' instead of 'VXI11' in the lxi_connect() call.RAW is faster because it does not suffer the overhead of the quite heavy VXI11 (SUN OneRPC) protocol.Soon liblxi will also support HiSLIP which basically brings the same speed as RAW but with the benefit of some instrument relevant protocol features.I plan to release the HiSLIP feature start next year.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 19, 2022, 09:34:37 am
content: Tried liblxi with TCP RAW, and I don't know how to handle the transfers.- it puts consecutive lxi_send() strings in the same data packet, which the oscilloscope won't understand. I don't know how to flush the transmit buffer after each lxi_send(), so as a workaround I'm adding an *OPC? at the end of each command.- even so, when the oscilloscope sends the first 250 000 data bytes, the lxi_receive() returns only the payload of the first TCP packet, which is 1460 bytes, while the oscilloscope continues to send more packets. I was expecting for the lxi_receive() to concatenate the payload of all the incoming TCP packets, and to stop only when the whole 250000 requested bytes are received (or if a timeout occurs), but lxi_receive() returns after the first TCP packet. Not sure if this is a bug or it's the NO_BLOCKING setting of a TCP RAW connection.I don't know how to send TCP_BLOCKING (or alike) to the socket, or if such a feature exists in a liblxi TCP RAW connection.Code:// liblxi git at https://github.com/lxi-tools/liblxi// to install it from the ubuntu repository (lxilib-dev is in most of the Linux distros repos and in FreeBSD):// sudo apt install liblxi-dev// headers in /usr/include/lxi.h// .so lib in /usr/lib/x86_64-linux-gnu/liblxi.so// display all installed files with 'dpkg -L liblxi-dev' or// sudo apt install apt-file// sudo apt-file update// (sudo???) apt-file list liblxi-dev// no need to append LIBRRY_PATH or LD_LIBRARY_PATH// man lxi_<TAB><TAB>// lxi_connect lxi_discover lxi_init lxi_send lxi_disconnect lxi_discover_if lxi_receive// compile and run this 'demo.c' from a Linux terminal (!!! -l options _must_ be positioned after the .c sources, not before)// cd /home/muuu/wd8TB/2019/Z/_hobby/__Pnnn/221103___VNA_DG4102_DS1054Z_python/sw/C// gcc -Werror -Wfatal-errors ./dl_24M_ADC_samples_ds1054z.c -llxi -ltirpc && time ./a.out#include <stdio.h>#include <stdlib.h>#include <string.h>#include <time.h>#include <lxi.h>int main(){printf("liblxi w DS1054Z using TCP mode RAW");// 12 bytes preamble + max of 24 mil bytes of ADC samples + 1 byte for the end of string terminator#define CHARS_BUFFER_MAX 24000000L // in specs max Rx in a single chunk is 250k chars for Rigol DS1054Zchar *rx24MB = NULL; // declare a pointer to char, and initialize it with NULLrx24MB = malloc(CHARS_BUFFER_MAX * sizeof *rx24MB); // allocated memory on the heap for rx24MBif (!rx24MB) { // !!! always check the memory was allocated successfully !!!fputs ("ERROR: allocation memory failed for the 'rx24MB' buffer, exiting.", stderr);exit (EXIT_FAILURE);}int device, length, timeout = 2000; // timeout is in mschar response; // allocated on the stack, segmentation fault if too large, Linux max stack is 8MiB//char *command = "*IDN?"; // with or without \n, the osc responded in a VXI11 connectionchar *command = "*IDN?\n";lxi_init(); // Initialize LXI library//device = lxi_connect("192.168.1.3", 0, "inst0", timeout, VXI11); // instument responded to either *IDN? or *IDN?\n// device = lxi_connect("192.168.1.3", 0, NULL, timeout, VXI11); // Connect to LXI devicedevice = lxi_connect("192.168.1.3", 5555, "inst0", timeout, RAW); // must end in \n, or else *IDN? timeout as incomplete// device = lxi_connect("192.168.1.3", 5555, "TCP", timeout, RAW); // *IDN? Timeout err for timeout 2000// device = lxi_connect("192.168.1.3", 0, "inst0", timeout, RAW); // NOPE// device = lxi_connect("192.168.1.3", 5555, "inst0", timeout, HISLIP); // *IDN? garbled chars for timeout 2000lxi_send(device, command, strlen(command), timeout); // Send SCPI command ("*IDN?")lxi_receive(device, response, sizeof(response), timeout); // Rx until "\n", or sizeoff(response) chars, or timeoutprintf("%s\n", response);// exit if no DS1054Z// exit(0);// enable only ch1command = ":CHAN1:DISP 1;*OPC?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout); // Rx until "\n"?, or sizeoff(response) chars, or timeoutcommand = ":CHAN2:DISP 0;*OPC?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);command = ":CHAN3:DISP 0;*OPC?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);command = ":CHAN4:DISP 0;*OPC?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);// ??? here to insert ":RUN""// set acquisition mode to max 24_000_000 points memory depth// "auto", 12K, 120K, 1.2M, 12M, 24M"long mdep = 24000000L;command = ":RUN;*OPC?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);// sprintf(command, ":ACQ:MDEP %li", mdep);// sprintf(command, ":ACQ:MDEP %li%s", mdep, "\0");command = ":ACQ:MDEP 24000000;*OPC?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);// wait for triggereddo {command = ":TRIGger:STATus?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);} while (response != 'T');// then wait for ADC to aquire enough dataclock_t stop_time;printf("wait %ld\n", clock());stop_time = clock() + 30000; //30 secondsprintf("beep %ld\n", stop_time);while (clock() < stop_time) ;printf("go %ld\n", clock());// go to stop mode and prepare to dl 24MSacommand = ":STOP;*OPC?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);command = ":WAV:SOUR CHAN1;*OPC?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);command = ":WAV:MODE RAW;*OPC?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);command = ":WAV:FORM BYTE;*OPC?\n";lxi_send(device, command, strlen(command), timeout);lxi_receive(device, response, sizeof(response), timeout);// char c2 = {0};char c2;// loop to dl all samples in chunks of (max) 250_000 samples// set start-stop indexes for next chunklong rx_len, chunk_len;// long chunk_start, chunk_stop, chunk_size = 125000L; //dl 24Msa/134slong chunk_start, chunk_stop, chunk_size = 250000L; //dl 24Msa/86s max from specs// long chunk_start, chunk_stop, chunk_size = 750000L; //dl 24Msa/65s// long chunk_start, chunk_stop, chunk_size = 1000000L; //dl 24Msa/56s preferred ???// long chunk_start, chunk_stop, chunk_size = 1175000L; //dl 24Msa/54slong samples_to_dl = 24000000L;if (samples_to_dl > mdep) {samples_to_dl = mdep;}for(chunk_start = 1; chunk_start < samples_to_dl; chunk_start += chunk_size) {sprintf(c2, ":WAV:STAR %li;*OPC?\n", chunk_start);puts(c2);rx_len = lxi_send(device, c2, strlen(c2), timeout);printf("ack_len=%ld\n", rx_len);lxi_receive(device, response, sizeof(response), timeout);//at the last chunk, chunk_len might be shorter than chunk_sizeif (chunk_start + chunk_size > samples_to_dl) {chunk_len = samples_to_dl - chunk_start + 1;chunk_stop = samples_to_dl;} else {chunk_len = chunk_size;chunk_stop = chunk_start + chunk_size - 1;}sprintf(c2, ":WAV:STOP %li;*OPC?\n", chunk_stop);puts(c2);rx_len = lxi_send(device, c2, strlen(c2), timeout);printf("ack_len=%ld\n", rx_len);lxi_receive(device, response, sizeof(response), timeout);command = ":WAV:DATA?\n";puts(command);rx_len = lxi_send(device, command, strlen(command), timeout);printf("ack_len=%ld\n", rx_len);rx_len = lxi_receive(device, (char *)(&rx24MB[chunk_start-1]), chunk_len, timeout);printf("rx_data_bytes=%ld\n", rx_len);int err = 0;if(rx_len != chunk_len) {printf("ERROR - received too few bytes: ");err = 1;}printf("chunk_len=%li rx_len=%li\n", chunk_len, rx_len);printf("\n");if(err) exit(err);}// Disconnectlxi_disconnect(device);}will output is this:Code:liblxi w DS1054Z using TCP mode RAWRIGOL TECHNOLOGIES,DS1104Z,DS1ZA164658712,00.04.05.SP2wait 15947beep 45970go 45970:WAV:STAR 1;*OPC?ack_len=18:WAV:STOP 250000;*OPC?ack_len=23:WAV:DATA?ack_len=11rx_data_bytes=1460ERROR - received too few bytes: chunk_len=250000 rx_len=1460

author: RoGeorge
author_rank: Super Contributor
timestamp: November 20, 2022, 12:17:00 pm
content: To learn what SOCKET parameters are optimal for a connection with the instrument, I had to read about TCP sockets, and realized that for LXI instruments it is trivial to send SCPI commands (and read the response) directly from Python, using its included 'socket' module.Code:import socketsocket.setdefaulttimeout(5000)with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:s.connect(('192.168.1.3', 5555))s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)s.sendall(b'*IDN?\n')ans = s.recv(1024)print(ans)Will printQuoteWhere:#192.168.1.3 # is the fixed IP of the instrument (a Rigol DS1054Z oscilloscope)# 5555 # PORT number (fixed by manufacturer) 5555 for Rigol, 5025 for Agilent, etc.# 5000 # the timeout, in ms# 1024 # the size of the receive bufferI only need to connect by LAN, using fixed IP, don't need any VXI11 or PyVISA or Telnet or any other dependency.

author: switchabl
author_rank: Frequent Contributor
timestamp: November 20, 2022, 01:37:12 pm
content: One thing to be aware of when using plain TCP sockets is that TCP is astream-orientedprotocol. Oh, of course itoperateson packets at the implementation level. But it tries to hide all of that away and provide you with a pipe where you can put in bytes at one end (send) and takesomebytes out again at the other end (recv). There is no concept of a "message" and onesendon the scope might not correspond to a singlerecvon your side.In other words, your call torecvis allowed to returnCode:b'RIGOL TECHNOLOGIES,DS1'or even justCode:b'R'The result will depend on the buffering strategies (and socket options) on either end as well as things like the underlying network packet size. So your example might not work on a different PC and it might not even work the next time you try it. You really need to keep callingrecvuntil you have the expected number of bytes (or the termination character or hit a timeout).In order to minimize latency, the other socket option you might want to try is TCP_QUICKACK (in addition to or instead of TCP_NODELAY).

author: lundmar
author_rank: Frequent Contributor
timestamp: November 20, 2022, 06:24:51 pm
content: As it has been hinted, TCP is a stream oriented protocol so talking packets etc. does not make much sense.When you use RAW mode with liblxi the interface behaves a bit more low level and you basically have to do two things when sending commands and potentially receiving large amounts of response data.1. Send command: Append newline ('\n') to command string so that receiving instrument knows a command has been received and proceeds to process it2. Receive response: Keep calling lxi_receive() until it returns error (timeout) which will be the only indication that the instrument has no more response data to send.That is the downside of using TCP/RAW - one must rely on and wait for timeout because there is no protocol telling the client how much data to expect from the server.This is exactly what protocols like VXI11 and HiSLIP takes care of. VXI11 does it in a bad way because of its heavy protocol overhead but HiSLIP solves that.Now, I could move some of that complexity from the example to the liblxi implementation but I decided against it to give more control to the user so they can decide on buffering strategy etc.I've added some example code here which demonstrates how to send a SCPI command to request screenshot image data from an instrument and receive all the image data:https://github.com/lxi-tools/liblxi/blob/master/test/receive-image-data.c

author: RoGeorge
author_rank: Super Contributor
timestamp: November 21, 2022, 10:14:54 am
content: Quote from: switchabl on November 20, 2022, 01:37:12 pmThanks for the hint. Searched today about TCP_QUICKACK, and it seems to be Linux specific. Other TCP stacks don't have the TCP_QUICKACK setting (e.g. the BSD one in FreeBSD or MacOS).QuoteSource:https://man7.org/linux/man-pages/man7/tcp.7.htmlQuote from: lundmar on November 20, 2022, 06:24:51 pmYou are too kind, thank you. I understand now the need for '\n' with lxilib in RAW mode, makes sense. Also it clarifies my question about consolidating consecutive SCPI commands before sending them:Quote from: RoGeorge on November 19, 2022, 09:34:37 am'liblxi' works just fine, no need to flush. The instrument will understand multiple commands from a single TCP packet, just that I was sending a wrong SCPI command, a bug in my code, sorry.

author: alm
author_rank: Super Contributor
timestamp: November 21, 2022, 05:57:25 pm
content: Quote from: RoGeorge on November 19, 2022, 12:04:57 amI'm not sure why this is happening. As far as I know these libraries should use plain TCP sockets, and not implement the Telnet protocol. I would be careful with the word "raw TCP sockets", becauseraw socketsis generally understood to mean something else in network programming.I just tested it with PyVISA, and it works as expected for me:I used this as a mock server returning data containing null characters:echo '\0ff\0' | nc -vl -p 1052And then succesfully retrieved this data:Code:>>> import pyvisa>>> rm = pyvisa.ResourceManager()>>> res = rm.open_resource('TCPIP0::127.0.0.1::1052::SOCKET')>>> res.write_termination = '\n'>>> res.read_termination = '\n'>>> data = res.read()>>> data'\x00ff\x00'You don't happen to have the read termination set to \00, have you? Because then I could understand this happening.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 22, 2022, 06:00:29 pm
content: Might have been something I was doing wrong. Can not reproduce that error any longer.

author: MiDi
author_rank: Frequent Contributor
timestamp: March 11, 2023, 04:29:14 pm
content: Had similar problems:On one PC the readout of 24MSamples took more than 40s and on another ~22s (Both Win10 latest updates, no TCP mods, but different python & module versions).Turning Nagle off in Win did not help - as was expected asPyVisa turns it off by default for the connection itself.Turning delayed ACK off in Win did improve the readout times a bit (~36s), but still slow.As it turned out changing the code did the trick to get ~22s on both PCs:instead single commands:Code:instrument.write(f":WAV:STAR {interval}")# start index of memoryinstrument.write(f":WAV:STOP {interval}")# stop index of memorydata += (instrument.query_binary_values(f":WAV:DATA?", datatype='B')) # get the data, B = unsigned charall commands at once did it:Code:data += (instrument.query_binary_values(f":WAV:STAR {interval}\n:WAV:STOP {interval}\n:WAV:DATA?", datatype='B')) # get the data, B = unsigned charFor reference the full source code used:Code:#!/usr/bin/env python# -*- coding: utf-8 -*-# developed & tested with Python 3.6, pyvisa 1.11.3# Rigol DS1000Z data acquisition#from ctypes.wintypes import FLOAT#from unicodedata import decimalimport pyvisa as visa# needs a backend e.g. NI-VISA or PyVISA-pyimport sysfrom time import time, sleepfrom matplotlib.ticker import EngFormatterfrom decimal import *def ef(value, places=None):return EngFormatter(sep="", places=places).format_eng(float(value)) #round(value, 4) - need to round in exp formdef crange(start,end,step):i = startwhile i < end-step+1:yield i, i+step-1i += stepyield i, enddef wait_ready(instrument):#instrument.write("*OPC")instrument.write("*WAI")ready = instrument.query("*OPC?").strip()#print(ready)while ready != "1":# never occured, needed?ready = instrument.query("*OPC?").strip()print(f"\n-------------------not ready: {ready}-----------------------")#passdef is_error(instrument):wait_ready(instrument)status = instrument.query("*ESR?").strip()if status not in ('0', '1'):wait_ready(instrument)return instrument.query(":SYST:ERR?").strip()#:SYSTem:ERRor[:NEXT]?else:return False# connect to scope# initialize scope##*RST p. 96 - Restore the instrument to the default state##*WAI p. 97 - Wait for the operation to finish (maybe not helpful as it is only for parallel execution of commands)##:ACQ:MDEP? p. 21 - Set or query the memory depth of the oscilloscope (namely the number of waveform points that can be stored in a single trigger sample)##:ACQ:TYPE? p. 22 - Set or query the acquisition mode of the oscilloscope##:ACQ:SRAT? p. 23 - Query the current sample rate. The default unit is Sa/s.#def connect(timeout = 10000, chunk_size = 20*1024):def connect():try:# Get the USB device, e.g. 'USB0::0x1AB1::0x0588::DS1ED141904883'rm = visa.ResourceManager()# "@py" for PyVISA-pyinstruments = rm.list_resources()# rm.list_resources('USB?*')print(instruments)usb = list(filter(lambda x: 'USB' in x, instruments))# TODO filter for ::DS1Zprint(usb)#if len(usb) != 1:#print(f"No or multiple instruments found: {instruments}")#sys.exit(-1)#from pyvisa.highlevel import ascii, single, double#instrument = rm.open_resource(usb)#instrument = rm.open_resource(f"TCPIP0::192.168.1.26::5555::SOCKET", write_termination = '\n', read_termination = '\n', timeout = timeout, chunk_size = chunk_size)instrument = rm.open_resource(f"TCPIP0::192.168.1.26::5555::SOCKET", write_termination = '\n', read_termination = '\n')#instrument = rm.open_resource("TCPIP::192.168.1.26::INSTR", write_termination = '\n', read_termination = '\n')#instrument.write_termination = '\n'#instrument.read_termination = '\n'#instrument = rm.open_resource("USB0::0x1AB1::0x04CE::DS1ZA172316530::INSTR")#instrument.encoding='latin1' # hack - better do ask raw#rm.timeout = timeout#rm.chunk_size = chunk_sizereturn instrumentexcept Exception as e:print(f"Cannot open instrument: {str(e)}")sys.exit(-1)def setup(instrument, timebase_scale, ch1_scale, mem_depth):#timebase_scale = Decimal("50")# timebase scale in s#ch1_scale = Decimal("1e-3")# set channel scale in units (V, A, W)#mem_depth = "24000000"# set memory depth in points: 12000, 120000, 1200000, 12000000, 24000000timebase_scale = Decimal(timebase_scale)# timebase scale in sch1_scale = Decimal(ch1_scale)# set channel scale in units (V, A, W)mem_depth = mem_depth# set memory depth in points: 12000, 120000, 1200000, 12000000, 24000000instrument.write(f"*CLS")# Clear event registers and error queuewait_ready(instrument)status = instrument.query(":TRIG:STAT?").strip()# Stop instrument: defined statewhile status != "STOP":instrument.write(":STOP")wait_ready(instrument)status = instrument.query(":TRIG:STAT?").strip()set_timebase_scale = Decimal(instrument.query(":TIM:SCAL?").strip())# set timebase scalewhile set_timebase_scale != timebase_scale:print(f"set_timebase_scale: {set_timebase_scale}, timebase_scale: {timebase_scale}")instrument.write(f":TIM:SCAL {timebase_scale}")wait_ready(instrument)set_timebase_scale = Decimal(instrument.query(":TIM:SCAL?").strip())print(f"Timebase scale: {ef(float(set_timebase_scale))}s/div")set_ch1_scale = Decimal(instrument.query(":CHAN1:SCAL?").strip())# query channel scalewhile set_ch1_scale != ch1_scale:print(f"set_ch1_scale: {set_ch1_scale}, ch1_scale: {ch1_scale}")instrument.write(f":CHAN1:SCAL {ch1_scale}")wait_ready(instrument)set_ch1_scale = Decimal(instrument.query(":CHAN1:SCAL?").strip())print(f"Ch1 scale: {ef(float(set_ch1_scale))}V/div")set_mem_depth = instrument.query(":ACQ:MDEP?")# memory depth = bytes to readif set_mem_depth != mem_depth:status = instrument.query(":TRIG:STAT?").strip()# Start instrument: needed for setting MDEPwhile status == ("STOP"):print(f"status: {status}")instrument.write(":RUN")wait_ready(instrument)status = instrument.query(":TRIG:STAT?").strip()while set_mem_depth != mem_depth:print(f"set_mem_depth: {set_mem_depth}, mem_depth: {mem_depth}")instrument.write(f":ACQ:MDEP {mem_depth}")wait_ready(instrument)set_mem_depth = instrument.query(":ACQ:MDEP?")print(f"Memory depth: {ef(float(set_mem_depth))}pts")trig_pos = set_timebase_scale * 6# horizontal position of trigger on left edge (no pre-trigger)set_trig_pos = Decimal(instrument.query(":TIM:OFFS?").strip())while set_trig_pos != trig_pos:print(f"set_trig_pos: {set_trig_pos}, trig_pos: {trig_pos}")instrument.write(f":TIM:OFFS {trig_pos}")wait_ready(instrument)set_trig_pos = Decimal(instrument.query(":TIM:OFFS?").strip())print(f"Trigger position: {ef(float(set_trig_pos))}s")# read data (p. 241)# :STOP - better use single and wait until finished (if that is possible)# :WAV:FORM WORD - are there values beyond 255? Else BYTE would be sufficient##S1. :STOP Set the instrument to STOP state (you can only read the waveform data in the internal memory when the oscilloscope is in STOP state)##S2. :WAV:SOUR CHAN1 Set the channel source to CH1##S3. :WAV:MODE RAW Set the waveform reading mode to RAW##S4. :WAV:FORM WORD Set the return format of the waveform data to WORD##Perform the first reading operation##S5. :WAV:STAR 1 Set the start point of the first reading operation to the first waveform point##S6. :WAV:STOP 125000 Set the stop point of the first reading operation to the 125000th waveform point##S7. :WAV:DATA? Read the data from the first waveform point to the 125000th waveform point##Perform the second reading operation##S8. :WAV:STAR 125001 Set the start point of the second reading operation to the 125001th waveform point##S9. :WAV:STOP 250000 Set the stop point of the second reading operation to the 250000th waveform point##S10. :WAV:DATA? Read the data from the 125001th waveform point to the 250000th waveform point##Perform the third reading operation##S11. :WAV:STAR 250001 Set the start point of the third reading operation to the 250001th waveform point##S12. :WAV:STOP 300000 Set the stop point of the third reading operation to the 300000th waveform point (the last point)##S13. :WAV:DATA? Read the data from the 250001th waveform point to the 300000th waveform point (the last point)# improved read data##S1. :SING p. 19 - Set the oscilloscope to the single trigger mode##S2. :TFOR p. 19 - Generate a trigger signal forcefully. This command is only applicable to the normal and single trigger modes (see the :TRIGger:SWEep command) and is equivalent to pressing the FORCE key in the trigger control area on the front panel.##S3. wait for scope to finish##S4. :WAV:SOUR CHAN1 Set the channel source to CH1##S5. :WAV:MODE RAW Set the waveform reading mode to RAW##S6. :WAV:FORM BYTE p. 239 - Set the return format of the waveform data to WORD|BYTE|ASCii (WORD: higher byte is always 0) - default: BYTEdef capture_data(instrument):force_trigger = Trueset_trig_swe = instrument.query(":TRIG:SWE?").strip()# query trigger sweepprint(f"set_trig_swe: {set_trig_swe}")while set_trig_swe == "SING":# reset single trigger (defined state)instrument.write(":TRIG:SWE AUTO")wait_ready(instrument)set_trig_swe = instrument.query(":TRIG:SWE?").strip()while set_trig_swe != "SING":# single triggerinstrument.write(":SING")wait_ready(instrument)set_trig_swe = instrument.query(":TRIG:SWE?").strip()print("measurement started")# TODO refactor# :TRIG:STAT? returns TD, WAIT, RUN, AUTO, or STOP# RUN (Pre-Trigger measurement) - WAIT (for trigger, may not be read out if triggered by signal) - TD (Post-Trigger) - STOP (measurement finished)#wait until measurement finishedstart_time = time()run_time = start_timecurrent_time = start_timewait_ready(instrument)# even with wait_ready sometimesstatus = instrument.query(":TRIG:STAT?").strip()# get trigger status: TD, WAIT, RUN, AUTO or STOPprev_status = statuswait_count = 0finished = Falsewhile not finished:if status != prev_status:print()run_time = time()current_time = time()if status in ("RUN", "TD"):# RUN/TD - measurement is runnningprint(f"{status} {current_time - run_time:7.3f}s ", end="\r")elif status == "WAIT":# WAIT - waiting for triggerprint(f"{status} {current_time - run_time:.3f}s")if force_trigger:wait_ready(instrument)instrument.write(":TFOR")# force trigger (may need multiple tries to start)wait_count += 1print(f'trigger forced {wait_count}x')elif status == "STOP":# end on STOP (measurement finished)finished = Trueprint(f"{status} {current_time - run_time:.3f}s")else:# AUTO - should not happenprint(f"{status} {current_time - run_time:7.3f}s ", end="\r")prev_status = statusstatus = instrument.query(":TRIG:STAT?").strip()# get trigger status: TD, WAIT, RUN, AUTO or STOP#sleep(0.01)current_time = time()print(f"measurement finished in {current_time - start_time:.3f}s")def read_data(instrument, ch: int = 1):mem_depth = int(instrument.query(":ACQ:MDEP?"))# memory depth# :WAV needs to be en bloc, at least between STAR & STOPinstrument.write(f":WAV:SOUR CHAN{int(ch)}")# data from channel 1-4wait_ready(instrument)instrument.write(":WAV:MODE RAW")# data from internal memorywait_ready(instrument)instrument.write(":WAV:FORM BYTE")# data as bytedata = []start_time = time()for interval in crange(1, mem_depth, 250_000):# 250_000 is max chunk size for bytes#instrument.write(f":WAV:STAR {interval}")# start index of memory#instrument.write(f":WAV:STOP {interval}")# stop index of memory#data += (instrument.query_binary_values(f":WAV:DATA?", datatype='B')) # get the data, B = unsigned chardata += (instrument.query_binary_values(f":WAV:STAR {interval}\n:WAV:STOP {interval}\n:WAV:DATA?", datatype='B')) # get the data, B = unsigned charprint(f"{len(data)/mem_depth:3.0%} memory read in {time() - start_time:.3f}s ({len(data)}/{mem_depth}) reading: {interval}:{interval}", end="\r")print()# debugprint(mem_depth)sps = float(instrument.query(":ACQ:SRAT?").strip())# query the current sample rate in Sa/sprint(f"SPS: {ef(sps)}")timebase_scale = float(instrument.query(":TIM:SCAL?").strip())# query timebase scaleprint(f"Timebase scale: {ef(timebase_scale)}s/div")# debug end# debug#instrument.write(":WAV:FORM ASCii")# data as ASCII#ascii_data = instrument.query(":WAV:DATA?")# get the data#print("ASCII:")#print(ascii_data[11:1000])#print("ASCII END")# debug endreturn data# scale data# formula for converting in V (p. 242): scaled value = (value - YORigin - YREFerence) x YINCrement##:WAV:YINC? # p. 244, return in scientific notation, see details##:WAV:YOR? # p. 244, return as integer, see details##:WAV:YREF? # p. 245, returns 127 always?, see details##:WAV:PRE? # p. 246, returns 10 waveform parameters separated by ",", see detailsdef scale_data(instrument, data):wait_ready(instrument)mem_depth = Decimal(instrument.query(":ACQ:MDEP?").strip()).normalize()# memory depthwait_ready(instrument)wf_parameters = instrument.query(":WAV:PRE?").strip().split(",")# get waveform parameters: <format>,<type>,<points>,<count>,<xincrement>,<xorigin>,<xreference>,<yincrement>,<yorigin>,<yreference>print(f"Waveform parameters: {wf_parameters}")while Decimal(wf_parameters).normalize() not in (mem_depth, mem_depth/2, mem_depth/3, mem_depth/4):wait_ready(instrument)wf_parameters = instrument.query(":WAV:PRE?").strip().split(",")print(f"Waveform parameters: {wf_parameters}")yincrement = float(wf_parameters)yorigin = float(wf_parameters)yreference = float(wf_parameters)scaled_data = []for byte in data:scaled_data.append((float(byte) - yorigin - yreference) * yincrement)return scaled_datadef save_data():passdef screenshot(instrument):sleep(2)# wait until all messages are gone (e.g. can operate now)start_time = time()screenshot = instrument.query_binary_values(":DISP:DATA? ON,OFF,PNG", datatype='B') # ON,OFF,PNG - BMP is a bit fasterstop_time = time()print(f"Screenshot took {stop_time - start_time:.3f}s")return screenshotdef view_image(image):import matplotlib.pyplot as pltimport matplotlib.image as mpimgimg = mpimg.imread(image)imgplot = plt.imshow(img)plt.show()def main():#global measurement_filenametimebase_scale = "1"# timebase scale in sch1_scale = "1e-3"# set channel scale in units (V, A, W)mem_depth= "24000000"# set memory depth in points: 12000, 120000, 1200000, 12000000, 24000000data_path = r"D:\Dokumente\Software-Projekte\Rigol DS1000Z data acquisition"data_dir = ""data_ext = "csv"data_file = "test"scope = connect()setup(scope, timebase_scale, ch1_scale, mem_depth)# TODO get from capture_data and read_data as returnprobe_ratio = Decimal(scope.query(":CHAN1:PROB?").strip()).normalize()ch1_scale_ef = ef(Decimal(ch1_scale), places=0)sps_ef = ef(Decimal(scope.query(":ACQ:SRAT?").strip()).normalize(), places=0)acq_mode = scope.query(":ACQ:TYPE?").strip()mem_depth_ef = ef(scope.query(":ACQ:MDEP?").strip(), places=0)acq_time_ef = ef(Decimal(timebase_scale)*12, places=0)for n in range(1, 2):capture_data(scope)data = read_data(scope)print()print(f"data length: {len(data)}")#print(data[:100])scaled_data = scale_data(scope, data)print(f"scaled data length: {len(scaled_data)}")#print(scaled_data[:100])data_file_tail = f"x{probe_ratio} {ch1_scale_ef}V {sps_ef}SPS {acq_mode} {mem_depth_ef}pts {acq_time_ef}s"data_file_full = f"{data_path}\{data_dir}\{data_file} {n:02} - {data_file_tail}"import csvwith open(f"{data_file_full}.{data_ext}", "w", newline="") as f:writer = csv.writer(f)for item in scaled_data:writer.writerow([format(item, ".4g")])#to reduce the file size further, limit number to significant decimal digits / resolution (8bit/10div) depending on channel range#import numpy as np##np_sd = np.asarray(scaled_data)#np.savetxt("D:\Dokumente\Software-Projekte\Rigol DS1000Z data acquisition\DS1000Z_DAQ.csv", scaled_data, fmt = "%.4g")#to reduce the file size further, limit number to significant decimal digits / resolution (8bit/10div) depending on channel rangeprint (f"{data_file_full}.{data_ext} written")image = screenshot(scope)#view_image(image)with open(f"{data_file_full}.png", "wb") as f:# write screenshot to filef.write(bytearray(image))print (f"{data_file_full}.png written")if __name__ == "__main__":main()For the curious, the relevant parts of the wireshark logs are attached.