Repository: NeuralEnsemble/python-neo

Issue:
SpikeIO does not load .smr
Dear Neo developers,
I can not load some of my .smr files with Spike2IO. I work with **python 3.5.2** and tried Neo versions **0.8.0** and **0.9.0.dev**. The error message is: “_IndexError: index 0 is out of bounds for axis 0 with size 0_”.
Note that I got the same error for some other  .smr files and could resolve this by using try_signal_grouping=False, however, the issue persists for some other .smr files. 
You can download an example file here: https://www.dropbox.com/s/hqgs4biuf5bm237/EP147-148-150-154_44d000.smr?dl=0
This file contains several channels (e.g. 'HCi1') of LFP recordings.

The code to reproduce the error:

from neo.io import Spike2IO
r = Spike2IO(filename='myfilename.smr',try_signal_grouping=False)

Thank you very much for your support.



Discussion:
samuelgarcia: Hi @MarinManuel. This is super cool and should help many users.
Thanks a lot.
samuelgarcia: I have push the file "Two-mice-bigfile-test000.smr" 
See https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/src/master/spike2

Could add it [here in neo/test/rawiotest](https://github.com/NeuralEnsemble/python-neo/blob/master/neo/test/rawiotest/test_spike2rawio.py#L16) and [here in neo/test/iotest](https://github.com/NeuralEnsemble/python-neo/blob/master/neo/test/iotest/test_spike2io.py#L21) and check that test are passing ?


JuliaSprenger: The tests are failing as `channel_names` are expected to be of type 'S' and not unicode. @samuelgarcia was there a reason for this not to be unicode?
There are two options now:
1) change the definition of the `ChannelIndex` class and make `channel_name` unicode. This affects  all ios based on the raw mechanism.
2) Hackily reading the `channel_names` as unicode and converting it to type `S`.

I would go for approach 2, since the `ChannelIndex` object will anyway be replaced in the future (#817), so we don't need to introduce unnecessary changes for `ChannelIndex` objects now.

MarinManuel: @JuliaSprenger @samuelgarcia  I've added a line to convert back the unicode strings to ascii for the time being. Hopefully that does not introduce any more problems.
samuelgarcia: Hi all.
@JuliaSprenger this is strange. I thought we fix this old (and very anoying) behavior where everything was ascii string because of python 2. Now python27 is no more here and evrything should be unicode everwhere.

@MarinManuel : I propose to fix this str vs unicode problem outside this PR very very soon. So you could rebase this PR with the offcial fix. is it OK ?




JuliaSprenger: @MarinManuel the format change is in master now, could you rebase and undo your changes in `basefromrawio.py`?
samuelgarcia: @JuliaSprenger :Could we first merge this PR before  #775 which is also related to spike2io ?
JuliaSprenger: Sure, then we keep #775 in the queue.
samuelgarcia: In fact I ask exactly the contary I wanted.
I wanted to say:
"Could we merge first  #775 (mine older) and then #824 (this one)"

Sorry. One day I will read what I am writting before push the green button.


MarinManuel: @samuelgarcia Hi, I have reverted the commit on files channelindex.py and basefromrawio.py, is that good enough? Bit of a git newbie here...
JuliaSprenger: Hi @MarinManuel,
there are still conflicts between the current `master` branch on Neuralensemble and the master branch on your fork. What you can do is rebase your branch on top of the current Neuralensemble master. First make sure you also have an up-to-date local version of that one on your machine by running
`git fetch upstream` (assuming upstream points to Neuralensemble)
Then you can do the rebasing via
`git rebase upstream/master` - this will lead to the conflicts pointed out above to appear during the rebase  in your local files. The conflicts are indicated in the files as
```
<<<<
first set of changes
=======
second set of changes
>>>>>
```
Please edit the files to include all the changes that are suggested by Neuralensemble master and additionally keep the ones you want to include in this PR.
Finally you add the files using `git add <filename>` and run `git rebase --continue` to finalize the rebase process.
To update this PR you then push the changes you made to your fork: `git -f push origin master` (assuming origin points to your fork). Here you need the `-f` flag as you want to overwrite the history of the commits in your fork, which were changed during the rebase process.

I hope this helps. Tell us if there's issues during the process.

Patch:
diff --git a/neo/rawio/spike2rawio.py b/neo/rawio/spike2rawio.py
index 2e315a49..6ff64939 100644
--- a/neo/rawio/spike2rawio.py
+++ b/neo/rawio/spike2rawio.py
@@ -52,6 +52,10 @@ class Spike2RawIO(BaseRawIO):
                 info['dtime_base'] = 1e-6
                 info['datetime_detail'] = 0
                 info['datetime_year'] = 0
+            if info['system_id'] == 9:
+                self._diskblock = DISKBLOCK
+            else:
+                self._diskblock = 1
 
             self._time_factor = info['us_per_time'] * info['dtime_base']
 
@@ -83,7 +87,7 @@ class Spike2RawIO(BaseRawIO):
                 if chan_info['blocks'] == 0:
                     chan_info['t_start'] = 0.  # this means empty signals
                 else:
-                    fid.seek(chan_info['firstblock'])
+                    fid.seek(chan_info['firstblock'] * self._diskblock)
                     block_info = read_as_dict(fid, blockHeaderDesciption)
                     chan_info['t_start'] = float(block_info['start_time']) * \
                         float(info['us_per_time']) * float(info['dtime_base'])
@@ -97,12 +101,12 @@ class Spike2RawIO(BaseRawIO):
         self._by_seg_data_blocks = {}
         for chan_id, chan_info in enumerate(self._channel_infos):
             data_blocks = []
-            ind = chan_info['firstblock']
+            ind = chan_info['firstblock'] * self._diskblock
             for b in range(chan_info['blocks']):
                 block_info = self._memmap[ind:ind + 20].view(blockHeaderDesciption)[0]
                 data_blocks.append((ind, block_info['items'], 0,
                                     block_info['start_time'], block_info['end_time']))
-                ind = block_info['succ_block']
+                ind = block_info['succ_block'] * self._diskblock
 
             data_blocks = np.array(data_blocks, dtype=[(
                 'pos', 'int32'), ('size', 'int32'), ('cumsum', 'int32'),
@@ -680,3 +684,5 @@ dict_kind = {
     8: 'TextMark',
     9: 'RealWave',
 }
+
+DISKBLOCK = 512


Test patch:
diff --git a/neo/test/iotest/test_spike2io.py b/neo/test/iotest/test_spike2io.py
index 56cf8d5c..2c853a78 100644
--- a/neo/test/iotest/test_spike2io.py
+++ b/neo/test/iotest/test_spike2io.py
@@ -18,6 +18,7 @@ class TestSpike2IO(BaseTestIO, unittest.TestCase, ):
         'File_spike2_3.smr',
         '130322-1LY.smr',  # this is for bug 182
         'multi_sampling.smr',  # this is for bug 466
+        'Two-mice-bigfile-test000.smr',  # SONv9 file
     ]
     files_to_download = files_to_test
 
diff --git a/neo/test/rawiotest/test_spike2rawio.py b/neo/test/rawiotest/test_spike2rawio.py
index b65ff52e..a02a23e1 100644
--- a/neo/test/rawiotest/test_spike2rawio.py
+++ b/neo/test/rawiotest/test_spike2rawio.py
@@ -13,6 +13,7 @@ class TestSpike2RawIO(BaseTestRawIO, unittest.TestCase, ):
         'File_spike2_3.smr',
         '130322-1LY.smr',  # this is for bug 182
         'multi_sampling.smr',  # this is for bug 466
+        'Two-mice-bigfile-test000.smr',  # SONv9 file
     ]
     entities_to_test = files_to_download