Repository: biolab/orange3

Issue:
FR: Single-Layer Neural Network possible?
I could be misreading NN widget help file [1], but is it possible to specify a single-layer NN - i.e. no hidden layers? If I enter 0 for the number of hidden layers neurons, it gives an error (screenshot no 1), while if I leave it blank, it defaults to 10. As an example, for a NN with 4 inputs, one output with logistic, what would I need to input in the neurons count box to get it (screenshot no 2)?

![image](https://user-images.githubusercontent.com/32625394/152094575-c5fffdb2-00d1-4d7f-930a-6de602b94b66.png) ![image](https://user-images.githubusercontent.com/32625394/152094061-2da1ebca-4ec8-4981-ab70-98c2e32656cf.png)

[1] Orange Data Mining - Neural Network Widget
https://orangedatamining.com/widget-catalog/model/neuralnetwork/

Env: Windows + Orange 3.31


Discussion:
No, it's not possible, unfortunately. Orange uses scikit learn's [Multilayer Perceptron](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier), which doesn't allow no hidden layers.
@dsanalytics, what is your goal? To be able to demonstrate that no-hidden-layer NN is equal to log reg?
@markotoplak I know that already. I guess sklearn devs know better then anyone else how Data Science or Algorithms R&D are supposed to be done. Saying in a paper: "Dear reader, I cannot have single layer NN to show you, but trust me, it's not a big deal as sklearn devs know that nobody in the world is using those anymore", does not look good.
Then don't say it.

Alternatively, explain that NN without a hidden layer is just logistic regression.
> Saying: "Dear reader, I cannot have single layer NN to show you, but trust me, it's not a big deal as sklearn devs know that nobody in the world is using those anymore", does not look good.

I don't know what this is precisely referring to, but if it is to my comment, then you are misunderstanding how Orange is structured. We base our algorithms on sklearn as it is the best supported ML library for Python in existence. We will not be writing our own algorithms for NN just because you think sklearn's are not good enough for you. Your beef is with sklearn - write to them and argue for your use case. When they support it, we will. 
@ajdapretnar that was to Marko's comment. I know that it's sklearn's fault and that you rely on it - hence I did not blame Orange.
It is not their fault. Calling a NN without hidden layer a NN is misleading. If somebody talks about NN with no hidden layers, I'd have to think for a while, then I'd ask him if he means logistic regression. And attempting to construct a NN without hidden layers is almost certainly by miskake, so it's better to report it as such.

What do you call polynomial regression with powers up to 0? I would prefer to call it "the average".

What do you call a classification tree with a depth of 0? I would prefer calling it "majority classifier".

What do you, then, call a NN without hidden layers?

A motor bike without a motor is just a bike.
Khm, I just checked, and `sklearn.neural_network.MLPClassifier(hidden_layer_sizes=())` worked for me. So scikit-learn` does not limit it - Orange does. :)

I vote for allowing the widget to work without hidden layers. If nothing else, it makes demonstration of similarities between NN without hidden layers and other methods easier. That could be useful in an education setting.
I nevertheless don't like it.

What about the training? It does not work the same; does it (usually?) give the same result?
@janezd I do not want to get into a debate here too, but I disagree. SLNN is not the same as LR even with sigmoid as training differs: backprop vs max likelihood. But I'm sure you know all this professor.

@markotoplak great find!
> What about the training? It does not work the same; does it (usually?) give the same result?

Yep, the training is different and has more options. For `iris` it runs out of iterations by default, so it only managed to split class 0 from 1 and 2. But even that is interesting, because it shows how fitting the same model in different ways may present different outcomes.

Also, the regularization parameter works differently in between these two: `alpha` versus `C`.

But still, why not allow it? The only thing to modify (almost) is not to auto-fill the empty edit box.
>  SLNN is not the same as LR even with sigmoid as training differs: backprop vs max likelihood.

@dsanalytics, the training could be the same. NN backprop simplifies exactly to the same rules as when using gradient descent for logistic regression. It is just that usually libraries have different optimizers fro both problems.
@markotoplak I was testing it with
`clf = MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(0,), random_state=1)`
Which probably means one cannot pass it explicit 0, but an empty edit box might work.
> @markotoplak I was testing it with `clf = MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(0,), random_state=1)` 

This means 1 hidden later with 0 neurons. No wonder it did not work. :)



Right, my bad. In this case, I suppose we should support it.
Ugh, I still think it's a bad idea, but OK if you think we must. Perhaps at least with a warning or information that it's better to use LR?

Resolving patch:
diff --git a/Orange/widgets/model/owneuralnetwork.py b/Orange/widgets/model/owneuralnetwork.py
--- a/Orange/widgets/model/owneuralnetwork.py
+++ b/Orange/widgets/model/owneuralnetwork.py
@@ -14,6 +14,7 @@
 from Orange.data import Table
 from Orange.modelling import NNLearner
 from Orange.widgets import gui
+from Orange.widgets.widget import Msg
 from Orange.widgets.settings import Setting
 from Orange.widgets.utils.owlearnerwidget import OWBaseLearner
 
@@ -81,11 +82,11 @@ class OWNNLearner(OWBaseLearner):
     activation_index = Setting(3)
     solver_index = Setting(2)
     max_iterations = Setting(200)
-    alpha_index = Setting(0)
+    alpha_index = Setting(1)
     replicable = Setting(True)
-    settings_version = 1
+    settings_version = 2
 
-    alphas = list(chain([x / 10000 for x in range(1, 10)],
+    alphas = list(chain([0], [x / 10000 for x in range(1, 10)],
                         [x / 1000 for x in range(1, 10)],
                         [x / 100 for x in range(1, 10)],
                         [x / 10 for x in range(1, 10)],
@@ -94,6 +95,11 @@ class OWNNLearner(OWBaseLearner):
                         range(100, 200, 10),
                         range(100, 1001, 50)))
 
+    class Warning(OWBaseLearner.Warning):
+        no_layers = Msg("ANN without hidden layers is equivalent to logistic "
+                        "regression with worse fitting.\nWe recommend using "
+                        "logistic regression.")
+
     def add_main_layout(self):
         # this is part of init, pylint: disable=attribute-defined-outside-init
         form = QFormLayout()
@@ -184,10 +190,10 @@ def get_learner_parameters(self):
                 ("Replicable training", self.replicable))
 
     def get_hidden_layers(self):
+        self.Warning.no_layers.clear()
         layers = tuple(map(int, re.findall(r'\d+', self.hidden_layers_input)))
         if not layers:
-            layers = (10,)
-            self.hidden_layers_input = "10,"
+            self.Warning.no_layers()
         return layers
 
     def update_model(self):
@@ -306,6 +312,8 @@ def migrate_settings(cls, settings, version):
             if alpha is not None:
                 settings["alpha_index"] = \
                     np.argmin(np.abs(np.array(cls.alphas) - alpha))
+        elif version < 2:
+            settings["alpha_index"] = settings.get("alpha_index", 0) + 1
 
 
 if __name__ == "__main__":  # pragma: no cover


Test patch:
diff --git a/Orange/widgets/model/tests/test_owneuralnetwork.py b/Orange/widgets/model/tests/test_owneuralnetwork.py
--- a/Orange/widgets/model/tests/test_owneuralnetwork.py
+++ b/Orange/widgets/model/tests/test_owneuralnetwork.py
@@ -15,9 +15,24 @@ def setUp(self):
 
     def test_migrate_setting(self):
         settings = dict(alpha=2.9)
-        OWNNLearner.migrate_settings(settings, None)
+        OWNNLearner.migrate_settings(settings, 0)
         self.assertEqual(OWNNLearner.alphas[settings["alpha_index"]], 3)
 
         settings = dict(alpha=103)
-        OWNNLearner.migrate_settings(settings, None)
+        OWNNLearner.migrate_settings(settings, 0)
         self.assertEqual(OWNNLearner.alphas[settings["alpha_index"]], 100)
+
+        settings = dict(alpha_index=0)
+        OWNNLearner.migrate_settings(settings, version=1)
+        self.assertEqual(OWNNLearner.alphas[settings["alpha_index"]], 0.0001)
+
+    def test_no_layer_warning(self):
+        self.assertFalse(self.widget.Warning.no_layers.is_shown())
+
+        self.widget.hidden_layers_input = ""
+        self.widget.apply_button.button.click()
+        self.assertTrue(self.widget.Warning.no_layers.is_shown())
+
+        self.widget.hidden_layers_input = "10,"
+        self.widget.apply_button.button.click()
+        self.assertFalse(self.widget.Warning.no_layers.is_shown())