Repository: microsoft/Qcodes

Issue:
Matplotlib can't handle more than 499 data points 
Bug from Thorvald at qdev.


### Steps to reproduce
```python
# VNA power scan:
v1.spec_mode(0)
v1.bandwidth(500)
v1.avg(5)

a = 6.61064e9;
v1.start(a-20e6)
v1.stop(a+20e6)
v1.npts(101)

loop = qc.Loop(v1.power.sweep(-45,-55,1)).each(v1.trace)
data = loop.get_data_set();
plot=qc.MatPlot(array, subplots=(1,2))
plot.add(array, subplot=2)
loop.with_bg_task(plot.update, 0.0005).run()
```

### Expected behaviour
The plot should show the data

### Actual behaviour
The plot only show the data if a row has less than 499 points

### System
windows 

**qcodes branch**
master

**qcodes commit**
3475e49d95c587384102d754975a460089c5b75b


Discussion:
nb (from Thorvald) this was after a MatPlot update 1.5.3 although hadn't tried with this many points before the update). On downgrading to 1.5.1 still broken...

Is this specific to one instrument. 

I can't reproduce it doing something simple with a mock instrument (Slightly modified from the base tutorial producing a 600 time 650 plot in the end. 


```python
%matplotlib nbagg
import matplotlib.pyplot as plt
from pprint import pprint
import time
import numpy as np
import qcodes as qc

# spawn doesn't like function or class definitions in the interpreter
# session - had to move them to a file.
from toymodel import AModel, MockGates, MockSource, MockMeter, AverageGetter, AverageAndRaw

# now create this "experiment", note that all these are instruments 
model = AModel()
gates = MockGates('gates', model=model)
source = MockSource('source', model=model)
meter = MockMeter('meter', model=model)

# The station is a container for all instruments that makes it easy 
# to log meta-data
station = qc.Station(gates, source, meter)

# it's nice to have the key parameters be part of the global namespace
# that way they're objects that we can easily set, get, and slice
c0, c1, c2, vsd = gates.chan0, gates.chan1, gates.chan2, source.amplitude

loop = qc.Loop(c1[-30:30:0.1], 0.01).loop(c0[-30:35:.1], 0.00001).each(
    meter.amplitude, # first measurement, at c2=0 -> amplitude_0 bcs it's action 0
    qc.Task(c2.set, 1), # action 1 -> c2.set(1)
    meter.amplitude, # second measurement, at c2=1 -> amplitude_2 bcs it's action 2
    qc.Task(c2.set, 0)
    )
data = loop.get_data_set(name='2D_test')

plot = qc.MatPlot(data.meter_amplitude_0, cmap=plt.cm.hot, figsize=(12, 4.5), subplots=(1, 2))
plot.add(data.meter_amplitude_2, cmap=plt.cm.hot, subplot=2)
data2 = loop.with_bg_task(plot.update, plot.save,  0.0000001).run()
```
nope, we get it with alazar and VNA measurements on two different computers (I don't have great confidence in the version control on either computer but they used different install methods so that's at least 2 data points :P). Both live plot and plotting old data.  Didn't try it out with the tutorial notebook but I can try running the above code when I have a moment and see how it looks. 
That would be great. I am happy to test out the instrument too just need to hook it up or use the lab computer 
I think you'll have trouble getting hold of either of those instruments to be honest :P I guess it could be something to do with the 'hard parameter' aspect of it but that seems a bit obscure. PyQt plot has previously had a 'nonlinearity of setpoints' issue but its not obvious to me that it's linked. I'll let you know when I've tested the tutorial. It will late afternoon/tomorrow morning 

Investigating this a bit more the issue seems to be that the frequency setpoint array being filed with NaNs

In the case of 559 the frequency array is filled with NaNs rather than the actual frequency after the first 3 measurements. I bet this is the reason for #397 too
I.e. do something like:
```
import qcodes as qc
import matplotlib.pyplot as plt
loaded_data = qc.load_data("VNA_power_set_frequency.dat")
fig, ax = plt.subplots(1,2)
ax[0].plot(loaded_data.frequency[0])
ax[1].plot(loaded_data.frequency[3])
fig.tight_layout()
```
and notice that the second plot is empty because all the data is Not a Number (NaN)
yh that was something I noticed which sometimes happens he VNA driver (and promptly forgot about), it usually doesn't matter because its a setpoint array that we make up so will be the same array for each trace but I also have no clear idea at all why it sometimes doesnt woek considering that the driver should set these arrays for every trace... I have some doubts that it's the same as #397 because that can be 'solved' by increasing the limit for nonlinearity a bit as @MerlinSmiles suggested which I wouldn't expect to allow it to plot if it was because of NaN problems, also if it was that then why would it be ok live plotting it but not loading it in afterwards? I can look around briefly for a data set without NaNs though and see if it can plot that though as a check for that being the reason for this bug
This can be reproduces relatively simply with a mock instrument 


```python
%matplotlib nbagg
import matplotlib.pyplot as plt
import numpy as np
import qcodes as qc
from toymodel import AModel, MockGates, MockSource, MockMeter, AverageAndRaw, ArrayGetter

# now create this "experiment", note that all these are instruments 
model = AModel()
gates = MockGates('gates', model=model)
source = MockSource('source', model=model)
meter = MockMeter('meter', model=model)

# The station is a container for all instruments that makes it easy 
# to log meta-data
station = qc.Station(gates, source, meter)

# it's nice to have the key parameters be part of the global namespace
# that way they're objects that we can easily set, get, and slice
c0, c1, c2, vsd = gates.chan0, gates.chan1, gates.chan2, source.amplitude
loop = qc.Loop(c0.sweep(0,20,0.1), delay=0.001).each(ArrayGetter(meter.amplitude, c2[-10:10:0.2], 0.001))
data = loop.get_data_set(name='testsweep')
plot = qc.MatPlot()
plot.add(data.meter_amplitude)
_ = loop.with_bg_task(plot.update).run()
plot2 = qc.MatPlot()
plot2.add(data.meter_amplitude)
```
However it only happens with the background task updating a plot. Removing the background task or replacing it with a dumme task seems to fix the issue. 
So the problem is that the update function calls `dataset.sync` which ends up rereading the datafile from disk. This overwrites the dataset which already has all the hard arraylike setpoints defined with the on disk version which only has the setpoint arrays for already measured values. I think this can be work around by passing a dummy update function to `plot.add`. 

Something like 

```
def dummy_sync():
    pass

%matplotlib nbagg
import matplotlib.pyplot as plt
import numpy as np
import qcodes as qc
from toymodel import AModel, MockGates, MockSource, MockMeter, AverageAndRaw, ArrayGetter

# now create this "experiment", note that all these are instruments 
model = AModel()
gates = MockGates('gates', model=model)
source = MockSource('source', model=model)
meter = MockMeter('meter', model=model)

# The station is a container for all instruments that makes it easy 
# to log meta-data
station = qc.Station(gates, source, meter)

# it's nice to have the key parameters be part of the global namespace
# that way they're objects that we can easily set, get, and slice
c0, c1, c2, vsd = gates.chan0, gates.chan1, gates.chan2, source.amplitude
loop = qc.Loop(c0.sweep(0,20,0.1), delay=0.001).each(ArrayGetter(meter.amplitude, c2[-10:10:0.2], 0.001))
data = loop.get_data_set(name='testsweep')
plot = qc.MatPlot()
plot.add(data.meter_amplitude, updater=dummy_sync)
_ = loop.with_bg_task(plot.update).run()
plot2 = qc.MatPlot()
plot2.add(data.meter_amplitude)
```
does not reproduce the issue
@jenshnielsen, thanks for this. Do you have ideas for a 'non workaround' fix?
We agreed to remove the sync from the default update function. I will make that change today and we can push a new release with that fix soon. 

Resolving patch:
diff --git a/docs/examples/toymodel.py b/docs/examples/toymodel.py
--- a/docs/examples/toymodel.py
+++ b/docs/examples/toymodel.py
@@ -4,7 +4,7 @@
 
 from qcodes import MockInstrument, MockModel, Parameter, Loop, DataArray
 from qcodes.utils.validators import Numbers
-
+from qcodes.instrument.mock import ArrayGetter
 
 class AModel(MockModel):
     def __init__(self):
@@ -147,36 +147,3 @@ def get(self):
         array = data.arrays[self.measured_param.full_name]
         return (array, array.mean())
 
-
-class ArrayGetter(Parameter):
-    """
-    Example parameter that just returns a single array
-
-    TODO: in theory you can make this same Parameter with
-    name, label & shape (instead of names, labels & shapes) and altered
-    setpoints (not wrapped in an extra tuple) and this mostly works,
-    but when run in a loop it doesn't propagate setpoints to the
-    DataSet. We could track down this bug, but perhaps a better solution
-    would be to only support the simplest and the most complex Parameter
-    forms (ie cases 1 and 5 in the Parameter docstring) and do away with
-    the intermediate forms that make everything more confusing.
-    """
-    def __init__(self, measured_param, sweep_values, delay):
-        name = measured_param.name
-        super().__init__(names=(name,))
-        self._instrument = getattr(measured_param, '_instrument', None)
-        self.measured_param = measured_param
-        self.sweep_values = sweep_values
-        self.delay = delay
-        self.shapes = ((len(sweep_values),),)
-        set_array = DataArray(parameter=sweep_values.parameter,
-                              preset_data=sweep_values)
-        self.setpoints = ((set_array,),)
-        if hasattr(measured_param, 'label'):
-            self.labels = (measured_param.label,)
-
-    def get(self):
-        loop = Loop(self.sweep_values, self.delay).each(self.measured_param)
-        data = loop.run_temp()
-        array = data.arrays[self.measured_param.full_name]
-        return (array,)
diff --git a/qcodes/data/data_set.py b/qcodes/data/data_set.py
--- a/qcodes/data/data_set.py
+++ b/qcodes/data/data_set.py
@@ -392,15 +392,7 @@ def sync(self):
         # could find a robust and intuitive way to make modifications to the
         # version on the DataServer from the main copy)
         if not self.is_live_mode:
-            # LOCAL DataSet - just read it in
-            # Compare timestamps to avoid overwriting unsaved data
-            if self.last_store > self.last_write:
-                return True
-            try:
-                self.read()
-            except IOError:
-                # if no files exist, they probably haven't been created yet.
-                pass
+            # LOCAL DataSet - no need to sync just use local data
             return False
             # TODO - for remote live plotting, maybe set some timestamp
             # threshold and call it static after it's been dormant a long time?
diff --git a/qcodes/instrument/mock.py b/qcodes/instrument/mock.py
--- a/qcodes/instrument/mock.py
+++ b/qcodes/instrument/mock.py
@@ -3,6 +3,9 @@
 from datetime import datetime
 
 from .base import Instrument
+from .parameter import Parameter
+from qcodes import Loop
+from qcodes.data.data_array import DataArray
 from qcodes.process.server import ServerManager, BaseServer
 from qcodes.utils.nested_attrs import _NoDefault
 
@@ -279,3 +282,36 @@ def _delattr(self, attr):
         See NestedAttrAccess for details.
         """
         self.ask('method_call', 'delattr', attr)
+
+class ArrayGetter(Parameter):
+    """
+    Example parameter that just returns a single array
+
+    TODO: in theory you can make this same Parameter with
+    name, label & shape (instead of names, labels & shapes) and altered
+    setpoints (not wrapped in an extra tuple) and this mostly works,
+    but when run in a loop it doesn't propagate setpoints to the
+    DataSet. We could track down this bug, but perhaps a better solution
+    would be to only support the simplest and the most complex Parameter
+    forms (ie cases 1 and 5 in the Parameter docstring) and do away with
+    the intermediate forms that make everything more confusing.
+    """
+    def __init__(self, measured_param, sweep_values, delay):
+        name = measured_param.name
+        super().__init__(names=(name,))
+        self._instrument = getattr(measured_param, '_instrument', None)
+        self.measured_param = measured_param
+        self.sweep_values = sweep_values
+        self.delay = delay
+        self.shapes = ((len(sweep_values),),)
+        set_array = DataArray(parameter=sweep_values.parameter,
+                              preset_data=sweep_values)
+        self.setpoints = ((set_array,),)
+        if hasattr(measured_param, 'label'):
+            self.labels = (measured_param.label,)
+
+    def get(self):
+        loop = Loop(self.sweep_values, self.delay).each(self.measured_param)
+        data = loop.run_temp()
+        array = data.arrays[self.measured_param.full_name]
+        return (array,)
\ No newline at end of file


Test patch:
diff --git a/qcodes/tests/test_loop.py b/qcodes/tests/test_loop.py
--- a/qcodes/tests/test_loop.py
+++ b/qcodes/tests/test_loop.py
@@ -13,6 +13,7 @@
 from qcodes.data.io import DiskIO
 from qcodes.data.data_array import DataArray
 from qcodes.data.manager import get_data_manager
+from qcodes.instrument.mock import ArrayGetter
 from qcodes.instrument.parameter import Parameter, ManualParameter
 from qcodes.process.helpers import kill_processes
 from qcodes.process.qcodes_process import QcodesProcess
@@ -79,7 +80,7 @@ def test_background_and_datamanager(self):
         # you can only do in-memory loops if you set data_manager=False
         # TODO: this is the one place we don't do quiet=True - test that we
         # really print stuff?
-        data = self.loop.run(location=self.location, background=True)
+        data = self.loop.run(location=self.location, background=True, data_manager=True)
         self.check_empty_data(data)
 
         # wait for process to finish (ensures that this was run in the bg,
@@ -118,6 +119,9 @@ def test_local_instrument(self):
         self.check_loop_data(data)
 
     def test_background_no_datamanager(self):
+        # We don't support syncing data from a background process
+        # if not using a datamanager. See warning in ActiveLoop.run()
+        # So we expect the data to be empty even after running.
         data = self.loop.run(location=self.location,
                              background=True,
                              data_manager=False,
@@ -127,7 +131,7 @@ def test_background_no_datamanager(self):
         self.loop.process.join()
 
         data.sync()
-        self.check_loop_data(data)
+        self.check_empty_data(data)
 
     def test_foreground_and_datamanager(self):
         data = self.loop.run(location=self.location, background=False,
@@ -202,6 +206,16 @@ def test_enqueue(self):
         loop.run_temp()
         self.assertFalse(hasattr(loop, 'process'))
 
+    def test_sync_no_overwrite(self):
+        # Test fix for 380, this tests that the setpoints are not incorrectly
+        # overwritten by data_set.sync() for this to happen with the original code
+        # the delay must be larger than the write period otherwise sync is a no opt.
+
+        loop = Loop(self.gates.chan1.sweep(0, 1, 1), delay=0.1).each(ArrayGetter(self.meter.amplitude,
+                                                                                 self.gates.chan2[0:1:1], 0.000001))
+        data = loop.get_data_set(name='testsweep', write_period=0.01)
+        _ = loop.with_bg_task(data.sync).run()
+        assert not np.isnan(data.chan2).any()
 
 def sleeper(t):
     time.sleep(t)