Repository: biolab/orange3

Issue:
MNIST training
I am wondering whether and how the well known MINIST dataset can be trained using Orange3. 

The following is tried:

- Loading a manually exported cvs data file into Orange3
  - it took some time, and many of feature types are incorrectly recognized as categorical instead of numerical. 

- Building a table file manually using python script widget, but then Orange3 went down. 


Discussion:
Thanks for the report.

Do you think it would help us replicate the problem and fix it (if necessary) if you sent the file that you tried to import? And fill the issue template, so that we know which OS and, even more important, which version of Orange you use? And tell us what you did in the Python Script Widget? And how did Orange go down? :)
Thank you for asking. 

Here is the info about the environment:

- Mac OSX 11.2.3, Quad core Intel Core i7, 16 GB memory
- Orange3 3.28.0, installed using the dmg installer,
- Anaconda was pre-installed, conda 4.9.2

and

- just started to use Orange3, but familiar with python.

Unfortunately I can't reproduce the proboem I had.
I don't remember exactly how I wrote the python script.
I just started to use Orange3 and  tried to mimic the explanation below in order to create table for MNIST from Numpy array:

https://orange3.readthedocs.io/projects/orange-data-mining-library/en/latest/reference/data.domain.html

The data was downloaded from the link below:

https://archive.ics.uci.edu/ml/machine-learning-databases/mnist-mld/

Using python script widget was not successful because it made Orange 3 go down. I don't remember exactly, but a bug report was sent to you, I think.

The second approach I tried was to load the saved csv file converted from a numpy array.  The loading process was successful although it took some time. To my surprise, the converted csv file was about 1.2 GB big, so everytime when I tried to do something Orange3 took horribly long time before it shows something. With frustration I just throwed everything away. 

The second approach was made again with another mnist file downloaded from:

http://yann.lecun.com/exdb/mnist/

This time the converted csv file was "only" 130MB big, and Orange3 could manage it to load although it takes several minutes for every step. But SVM model managed to make good predictions.

The size difference was probably because of the data types for images. The second one used int, the first one used float, I guess.

===

Here are my new questions. 

- Is it normal that Orange3 needs several minutes or more than 10 minutes to import 70,000 (28*28)-size images?

- Or is there another way to work directly with numpy array?
How did you load the data? In Python script, with the File widget or with CSV Import. (The latter is preferrable and faster.)

Which steps took minutes?

What do you mean by "directly working with numpy array"? Orange has its own data structure, which wraps numpy array and adds some data, like column names and types. If you'd like to work with numpy directly, you should do it in Python instead. Orange's SVM is actually from scikit-learn, and for reading of CSV files, it uses pandas.
Thank you for asking again.

I think I can answer your questions better when I know a simple way to use the MNIST data set.
Would you please explain how to load mnist data set into Orange3?

I don't care from which data repo. Several python libraries such as sklearn, tensorflow provides it. Or one can download it directly from below:

http://yann.lecun.com/exdb/mnist/



As far as I see, these files are in some proprietary format, so Orange cannot load them directly. You can use Python Script or an external script to put this into a different format. But even then, I don't know how to use them directly in Orange. That's why I'm asking what you actually did -- since you say it took a long time, you must had been doing something. :)
Ok, I explain you again. 

As for using Python Script: It was not successful for me to use it. I think I just wasn't familiar with it. 

It would be very helpful when there is a guide or a tutorial about writing a python script which transforms a numpy array or pandas dataframe into Table. Is there one?

The way I tried next was the following:

The data was loaded first as numpy array using python, and then converted into a csv file which was then loaded into Oragne3. But loading it to Orange3 took several minutes. The csv file was 130MB big. 


We will replace the default script (Hello world) with one that generates and outputs a numpy array.

Resolving patch:
diff --git a/Orange/widgets/data/owpythonscript.py b/Orange/widgets/data/owpythonscript.py
--- a/Orange/widgets/data/owpythonscript.py
+++ b/Orange/widgets/data/owpythonscript.py
@@ -36,6 +36,18 @@
 __all__ = ["OWPythonScript"]
 
 
+DEFAULT_SCRIPT = """import numpy as np
+from Orange.data import Table, Domain, ContinuousVariable, DiscreteVariable
+
+domain = Domain([ContinuousVariable("age"),
+                 ContinuousVariable("height"),
+                 DiscreteVariable("gender", values=("M", "F"))])
+arr = np.array([
+  [25, 186, 0],
+  [30, 164, 1]])
+out_data = Table.from_numpy(domain, arr)
+"""
+
 def text_format(foreground=Qt.black, weight=QFont.Normal):
     fmt = QTextCharFormat()
     fmt.setForeground(QBrush(foreground))
@@ -423,8 +435,8 @@ class Outputs:
 
     settings_version = 2
     scriptLibrary: 'List[_ScriptData]' = Setting([{
-        "name": "Hello world",
-        "script": "print('Hello world')\n",
+        "name": "Table from numpy",
+        "script": DEFAULT_SCRIPT,
         "filename": None
     }])
     currentScriptIndex = Setting(0)


Test patch: