Repository: pymeasure/pymeasure

Issue:
Error in examples/Notebook Experiments/script2.ipynb
script.ipynb runs fine but in script2.ipynb I hit the following error at `experiment = Experiment('test', procedure, analyse)`:

```python

C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\__init__.py in __setitem__(self, key, val)
    927             raise KeyError(
    928                 '%s is not a valid rc parameter. See rcParams.keys() for a '
--> 929                 'list of valid parameters.' % (key,))
    930 
    931     def __getitem__(self, key):

KeyError: 'axes.color_cycle is not a valid rc parameter. See rcParams.keys() for a list of valid parameters.'
```

Error in examples/Notebook Experiments/script2.ipynb
script.ipynb runs fine but in script2.ipynb I hit the following error at `experiment = Experiment('test', procedure, analyse)`:

```python

C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\__init__.py in __setitem__(self, key, val)
    927             raise KeyError(
    928                 '%s is not a valid rc parameter. See rcParams.keys() for a '
--> 929                 'list of valid parameters.' % (key,))
    930 
    931     def __getitem__(self, key):

KeyError: 'axes.color_cycle is not a valid rc parameter. See rcParams.keys() for a list of valid parameters.'
```



Discussion:
@robmarkcole, this looks like a matplotlib issue. Have you looked to see if `axes.color_cycle` is a valid RC parameter (a setting for matplotlib)? Can you provide the full traceback?
this has been deprecated since mpl 1.5 (https://matplotlib.org/api/api_changes.html#color-cycle-deprecated), and is now removed. Maybe [this](https://stackoverflow.com/questions/38045437/how-to-fix-py872-axes-color-cycle-is-deprecated-and-replaced-with-axes-prop-c) helps fixing the error.
Full traceback

```

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\__init__.py in __setitem__(self, key, val)
    921             try:
--> 922                 cval = self.validate[key](val)
    923             except ValueError as ve:

KeyError: 'axes.color_cycle'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-6-7599b72d4eb6> in <module>()
----> 1 experiment = Experiment('test', procedure, analyse)

C:\ProgramData\Anaconda3\lib\site-packages\pymeasure\experiment\experiment.py in __init__(self, title, procedure, analyse)
    115 
    116         config = get_config()
--> 117         set_mpl_rcparams(config)
    118         if 'Logging' in config._sections.keys():
    119             self.scribe = setup_logging(log, **config._sections['Logging'])

C:\ProgramData\Anaconda3\lib\site-packages\pymeasure\experiment\config.py in set_mpl_rcparams(config)
     71         import matplotlib
     72         for key in config._sections['matplotlib.rcParams']:
---> 73             matplotlib.rcParams[key] = eval(config._sections['matplotlib.rcParams'][key])

C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\__init__.py in __setitem__(self, key, val)
    927             raise KeyError(
    928                 '%s is not a valid rc parameter. See rcParams.keys() for a '
--> 929                 'list of valid parameters.' % (key,))
    930 
    931     def __getitem__(self, key):

KeyError: 'axes.color_cycle is not a valid rc parameter. See rcParams.keys() for a list of valid parameters.'
```
https://matplotlib.org/api/matplotlib_configuration_api.html#matplotlib.RcParams

`axes.color_cycle` was deprecated as you say https://matplotlib.org/users/prev_whats_new/whats_new_1.5.html?highlight=axes%20color_cycle#added-axes-prop-cycle-key-to-rcparams

As per the linked article I added `axes.prop_cycle : cycler('color', ['b', 'g', 'r', 'c', 'm', 'y', 'k'])` to my matplotlibrc file but error persists
@bilderbuchi, thanks for the deprecation reference!

@robmarkcole, is the new error occuring in the following section? I would suggest manually trying to set the RC param to ensure it is working as expected.
https://github.com/ralph-group/pymeasure/blob/4114450cb93f5bf435d771337bdcb40d79b0e748/pymeasure/experiment/config.py#L73
@cjermain I'm not sure what you mean, so I commented out `set_mpl_rcparams(config)` in `experiment.py` and the notebook runs fine with no errors
Ok, no worries. I was suggesting setting [`matplotlib.rcParams`](https://matplotlib.org/users/customizing.html#matplotlib-rcparams) directly to make sure that it works manually.

It would be great to put together a pull request that solves this. Any interest, @robmarkcole?


Hi @cjermain by resolve, do you mean my sollution of commenting out `set_mpl_rcparams`? Happy to contribute, and if we are given time to work on our automation project I will have a lot of equipment to add! Re live plotting, matplotlib isn't ideal for this, has Bokeh or similar been tested/ ruled out?
I think for live plotting aspects, pymeasure already uses pyqtgraph, not matplotlib.
also, the erroring entry is being created (I just now realise) in the scripty2.ipynb notebook itself, in the first input cell there is a %%writefile magic - the line `axes.color_cycle...` has to be adapted to the new matplotlib syntax, and the error should disappear.
I edited the first cell:
```
%%writefile my_config.ini
[Filename]
prefix = my_data_
dated_folder = 1
directory = data
ext = csv
index = 
datetimeformat = %Y%m%d_%H%M%S

[Logging]
console = 1
console_level = WARNING
filename = test.log
file_level = DEBUG

[matplotlib.rcParams]
axes.axisbelow = True
#axes.color_cycle = [(0.2980392156862745, 0.4470588235294118, 0.6901960784313725),
#                    (0.3333333333333333, 0.6588235294117647, 0.40784313725490196),
#                    (0.7686274509803922, 0.3058823529411765, 0.3215686274509804),
#                    (0.5058823529411764, 0.4470588235294118, 0.6980392156862745),
#                    (0.8, 0.7254901960784313, 0.4549019607843137),
#                    (0.39215686274509803, 0.7098039215686275, 0.803921568627451)]
axes.prop_cycle : cycler('color', ['b', 'g', 'r', 'c', 'm', 'y', 'k'])
```
And rerun but get the error:
```
C:\ProgramData\Anaconda3\lib\site-packages\pymeasure\experiment\experiment.py in __init__(self, title, procedure, analyse)
    115 
    116         config = get_config()
--> 117         set_mpl_rcparams(config)
    118         if 'Logging' in config._sections.keys():
    119             self.scribe = setup_logging(log, **config._sections['Logging'])

C:\ProgramData\Anaconda3\lib\site-packages\pymeasure\experiment\config.py in set_mpl_rcparams(config)
     71         import matplotlib
     72         for key in config._sections['matplotlib.rcParams']:
---> 73             matplotlib.rcParams[key] = eval(config._sections['matplotlib.rcParams'][key])

C:\ProgramData\Anaconda3\lib\site-packages\pymeasure\experiment\config.py in <module>()

NameError: name 'cycler' is not defined
```
I then added to `config.py` : `from cycler import cycler` and the error is fixed. So a PR will be required
@robmarkcole, nice work diagnosing the issue. A PR would be great!

As @bilderbuchi mentioned, the primary graphical user interfaces provided by PyMeasure use pyqtgraph for plotting. They provide reasonable performance, and are much faster than matplotlib. The notebook work was pioneered by @guenp, who also did some work with TCP instruments if I recall correctly -- she may be willing to share some insight on both the notebooks and TCP.
@robmarkcole, this looks like a matplotlib issue. Have you looked to see if `axes.color_cycle` is a valid RC parameter (a setting for matplotlib)? Can you provide the full traceback?
this has been deprecated since mpl 1.5 (https://matplotlib.org/api/api_changes.html#color-cycle-deprecated), and is now removed. Maybe [this](https://stackoverflow.com/questions/38045437/how-to-fix-py872-axes-color-cycle-is-deprecated-and-replaced-with-axes-prop-c) helps fixing the error.
Full traceback

```

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\__init__.py in __setitem__(self, key, val)
    921             try:
--> 922                 cval = self.validate[key](val)
    923             except ValueError as ve:

KeyError: 'axes.color_cycle'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-6-7599b72d4eb6> in <module>()
----> 1 experiment = Experiment('test', procedure, analyse)

C:\ProgramData\Anaconda3\lib\site-packages\pymeasure\experiment\experiment.py in __init__(self, title, procedure, analyse)
    115 
    116         config = get_config()
--> 117         set_mpl_rcparams(config)
    118         if 'Logging' in config._sections.keys():
    119             self.scribe = setup_logging(log, **config._sections['Logging'])

C:\ProgramData\Anaconda3\lib\site-packages\pymeasure\experiment\config.py in set_mpl_rcparams(config)
     71         import matplotlib
     72         for key in config._sections['matplotlib.rcParams']:
---> 73             matplotlib.rcParams[key] = eval(config._sections['matplotlib.rcParams'][key])

C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\__init__.py in __setitem__(self, key, val)
    927             raise KeyError(
    928                 '%s is not a valid rc parameter. See rcParams.keys() for a '
--> 929                 'list of valid parameters.' % (key,))
    930 
    931     def __getitem__(self, key):

KeyError: 'axes.color_cycle is not a valid rc parameter. See rcParams.keys() for a list of valid parameters.'
```
https://matplotlib.org/api/matplotlib_configuration_api.html#matplotlib.RcParams

`axes.color_cycle` was deprecated as you say https://matplotlib.org/users/prev_whats_new/whats_new_1.5.html?highlight=axes%20color_cycle#added-axes-prop-cycle-key-to-rcparams

As per the linked article I added `axes.prop_cycle : cycler('color', ['b', 'g', 'r', 'c', 'm', 'y', 'k'])` to my matplotlibrc file but error persists
@bilderbuchi, thanks for the deprecation reference!

@robmarkcole, is the new error occuring in the following section? I would suggest manually trying to set the RC param to ensure it is working as expected.
https://github.com/ralph-group/pymeasure/blob/4114450cb93f5bf435d771337bdcb40d79b0e748/pymeasure/experiment/config.py#L73
@cjermain I'm not sure what you mean, so I commented out `set_mpl_rcparams(config)` in `experiment.py` and the notebook runs fine with no errors
Ok, no worries. I was suggesting setting [`matplotlib.rcParams`](https://matplotlib.org/users/customizing.html#matplotlib-rcparams) directly to make sure that it works manually.

It would be great to put together a pull request that solves this. Any interest, @robmarkcole?


Hi @cjermain by resolve, do you mean my sollution of commenting out `set_mpl_rcparams`? Happy to contribute, and if we are given time to work on our automation project I will have a lot of equipment to add! Re live plotting, matplotlib isn't ideal for this, has Bokeh or similar been tested/ ruled out?
I think for live plotting aspects, pymeasure already uses pyqtgraph, not matplotlib.
also, the erroring entry is being created (I just now realise) in the scripty2.ipynb notebook itself, in the first input cell there is a %%writefile magic - the line `axes.color_cycle...` has to be adapted to the new matplotlib syntax, and the error should disappear.
I edited the first cell:
```
%%writefile my_config.ini
[Filename]
prefix = my_data_
dated_folder = 1
directory = data
ext = csv
index = 
datetimeformat = %Y%m%d_%H%M%S

[Logging]
console = 1
console_level = WARNING
filename = test.log
file_level = DEBUG

[matplotlib.rcParams]
axes.axisbelow = True
#axes.color_cycle = [(0.2980392156862745, 0.4470588235294118, 0.6901960784313725),
#                    (0.3333333333333333, 0.6588235294117647, 0.40784313725490196),
#                    (0.7686274509803922, 0.3058823529411765, 0.3215686274509804),
#                    (0.5058823529411764, 0.4470588235294118, 0.6980392156862745),
#                    (0.8, 0.7254901960784313, 0.4549019607843137),
#                    (0.39215686274509803, 0.7098039215686275, 0.803921568627451)]
axes.prop_cycle : cycler('color', ['b', 'g', 'r', 'c', 'm', 'y', 'k'])
```
And rerun but get the error:
```
C:\ProgramData\Anaconda3\lib\site-packages\pymeasure\experiment\experiment.py in __init__(self, title, procedure, analyse)
    115 
    116         config = get_config()
--> 117         set_mpl_rcparams(config)
    118         if 'Logging' in config._sections.keys():
    119             self.scribe = setup_logging(log, **config._sections['Logging'])

C:\ProgramData\Anaconda3\lib\site-packages\pymeasure\experiment\config.py in set_mpl_rcparams(config)
     71         import matplotlib
     72         for key in config._sections['matplotlib.rcParams']:
---> 73             matplotlib.rcParams[key] = eval(config._sections['matplotlib.rcParams'][key])

C:\ProgramData\Anaconda3\lib\site-packages\pymeasure\experiment\config.py in <module>()

NameError: name 'cycler' is not defined
```
I then added to `config.py` : `from cycler import cycler` and the error is fixed. So a PR will be required
@robmarkcole, nice work diagnosing the issue. A PR would be great!

As @bilderbuchi mentioned, the primary graphical user interfaces provided by PyMeasure use pyqtgraph for plotting. They provide reasonable performance, and are much faster than matplotlib. The notebook work was pioneered by @guenp, who also did some work with TCP instruments if I recall correctly -- she may be willing to share some insight on both the notebooks and TCP.

Resolving patch:
diff --git a/examples/Notebook Experiments/procedures.py b/examples/Notebook Experiments/procedures.py
deleted file mode 100644
--- a/examples/Notebook Experiments/procedures.py	
+++ /dev/null
@@ -1,60 +0,0 @@
-#
-# This file is part of the PyMeasure package.
-#
-# Copyright (c) 2013-2016 PyMeasure Developers
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-#
-
-import random
-from time import sleep
-from pymeasure.experiment import Procedure, IntegerParameter, Parameter, FloatParameter
-import logging
-log = logging.getLogger(__name__)
-log.addHandler(logging.NullHandler())
-
-class TestProcedure(Procedure):
-
-    iterations = IntegerParameter('Loop Iterations', default=100)
-    delay = FloatParameter('Delay Time', units='s', default=0.2)
-    seed = Parameter('Random Seed', default='12345')
-    
-    DATA_COLUMNS = ['Iteration', 'Random Number']
-
-    def startup(self):
-        log.info("Setting up random number generator")
-        random.seed(self.seed)
-
-    def execute(self):
-        log.info("Starting to generate numbers")
-        for i in range(self.iterations):
-            data = {
-                'Iteration': i,
-                'Random Number': random.random()
-            }
-            log.debug("Produced numbers: %s" % data)
-            self.emit('results', data)
-            self.emit('progress', 100.*i/self.iterations)
-            sleep(self.delay)
-            if self.should_stop():
-                log.warning("Catch stop command in procedure")
-                break
-
-    def shutdown(self):
-        log.info("Finished")
\ No newline at end of file
diff --git a/pymeasure/experiment/config.py b/pymeasure/experiment/config.py
--- a/pymeasure/experiment/config.py
+++ b/pymeasure/experiment/config.py
@@ -46,5 +46,6 @@ def get_config(filename='default_config.ini'):
 def set_mpl_rcparams(config):
     if 'matplotlib.rcParams' in config._sections.keys():
         import matplotlib
+        from cycler import cycler
         for key in config._sections['matplotlib.rcParams']:
             matplotlib.rcParams[key] = eval(config._sections['matplotlib.rcParams'][key])


Test patch: