Repository: mne-tools/mne-python

Issue:
BUG: kit data not readable anymore
Apparently some KIT data aren't readable any more, although they used to be:

```python
import mne
mne.io.read_raw_kit('R0972_Main1_3.7.16.sqd')
```
gives
```python
/Users/poeppellab/Applications/mne-python/mne/io/kit/kit.pyc in get_kit_info(rawfile)
    544         if version < 2 or (version == 2 and revision < 3):
    545             raise IOError("SQD file format V%iR%03i not supported." %
--> 546                           (version, revision))
    547 
    548         sysid = unpack('i', fid.read(KIT.INT))[0]

IOError: SQD file format V2R001 not supported.
```

but was running fine before, e.g. works ok on be75edaeabdd68628cf7bb7fed385deb8c966c2e

cc @teonbrooks @larsoner @christianbrodbeck 


Discussion:
data available here: https://wetransfer.com/downloads/43dd543c430906d9201f49b9b8aaf30520171002194824/551a677908caa8c7951ba80cd9453f8320171002194824/4813e6
Can you `git bisect` to find the problem?
I inserted that check when updating the SQD reader based on code from KIT. Since KIT had that check it seemed safer to include it. We could skip the check but risk unknown errors. Alternatively you can read and re-save the file with MEG160. 
Perhaps we should just emit a warning that the read might fail or that the data may be corrupted
+1 for warning, I suspect many of the KIT data acquired at NYU (New York) will have this issue, and we cant expect people to go to mne history and transform the sqd into fif
Is the data in question newly acquired? I guess that would mean your acquisition computer is running a very old version of MEG160. I thought ours here is ancient and its files can be read :)

To prevent people from missing warnings among other log messages we could raise an error unless a parameter like `unsupported_format` is set to `True`? Either is fine with me. I can make a PR.
I guess we do something similar for Neuromag data with `allow_maxshield=False` (error if not True), so I'm fine with `allow_unknown_format` or something similar. What do you think @kingjr ?
Actually, for the GUI we would need something like that, so we can watch for an exception and display a warning message.
> Is the data in question newly acquired?

Last year

> I guess that would mean your acquisition computer is running a very old version of MEG160. 

Looking at the acquisition server running on windows NT, I wouldn't be surprised...

> What do you think @kingjr ?

No strong preference, as long as I can read my data again.
Just a heads up, it's not going to be trivial, the old file apparently does not store gain information in the way we're now reading it... we will probably have to bring back the hard coded system-specific parameters.... 
@kingjr, is this the case for other files acquired on the NY system?
IDK, can you try it?

On 3 October 2017 at 13:21, Teon L Brooks <notifications@github.com> wrote:

> @kingjr <https://github.com/kingjr>, is this the case for other files
> acquired on the NY system?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mne-tools/mne-python/issues/4634#issuecomment-333916897>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AEp7DNMdns8cjjimM_LgD5seVGlKHwmfks5som0ngaJpZM4PrKOG>
> .
>

@christianbrodbeck tried it and it works when I set allow_unknown_format=True and then reads out "Force loading KIT format V2R001"
@avkasdan thanks! That's how I intended it, is that workable? Did you check that the data values are as expected?
They seem to be all there! Will confirm w @kingjr 
@kingjr do you have time to cross-check? Would be good to have for 0.15
I pulled the branch with @avkasdan, so +1 for merging

On 11 October 2017 at 09:23, Eric Larson <notifications@github.com> wrote:

> @kingjr <https://github.com/kingjr> do you have time to cross-check?
> Would be good to have for 0.15
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mne-tools/mne-python/issues/4634#issuecomment-335807919>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AEp7DILIIzJpY6RzbXqRd60TsE_8KfZYks5srMFpgaJpZM4PrKOG>
> .
>

I am really new to python, so how do I set allow_unknown_format=True when using kit2fiff. many thanks.
>
> mne.io.read_raw_kit(..., allow_unknown_format=True)
>

?

thank you.
this works when I use mne.io.read_raw_kit. But, I would like to use mne.gui.kit2fiff and convert sqd files to fiff. The data was collected a while ago so the format of sqd is probably the old one. 
I changed the default of allow_unknown_format in mne.io.kit to be True. Kit2fiff is working now for the old sqd files. 
Hope this won't cause any problems later
It should be pretty straight-forward to expose it as command line parameter, I will look into it. 

> On Sep 1, 2018, at 4:31 PM, XBT <notifications@github.com> wrote:
> 
> this works when I use mne.io.read_raw_kit. But, I would like to use mne.gui.kit2fiff and convert sqd files to fiff. The data was collected a while ago so the format of sqd is probably the old one.
> I changed the default of allow_unknown_format in mne.io.kit to be True. Kit2fiff is working now for the old sqd files.
> Hope this won't cause some problems later
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

@XBTinChina Actually, the GUI should already allow you to convert old files. If it does not, can you tell me what happens (or make available a sample file)?
https://github.com/mne-tools/mne-python/blob/master/mne/gui/_kit2fiff_gui.py#L309

Resolving patch:
diff --git a/mne/gui/_kit2fiff_gui.py b/mne/gui/_kit2fiff_gui.py
--- a/mne/gui/_kit2fiff_gui.py
+++ b/mne/gui/_kit2fiff_gui.py
@@ -416,7 +416,8 @@ def get_raw(self, preload=False):
                     stim_code, self.stim_threshold)
         raw = RawKIT(self.sqd_file, preload=preload, stim=self.stim_chs_array,
                      slope=self.stim_slope, stim_code=stim_code,
-                     stimthresh=self.stim_threshold)
+                     stimthresh=self.stim_threshold,
+                     allow_unknown_format=self.allow_unknown_format)
 
         if np.any(self.fid):
             raw.info['dig'] = _make_dig_points(self.fid[0], self.fid[1],


Test patch: