Repository: biolab/orange3

Issue:
'Normalizer' object has no attribute 'need_domain'
Hi,

I am testing out deploying my saved model in Python and am running into the following problem and can't quite figure out how to fix yet.  Any help would be greatly appreciated.

model = pickle.load(open("modelpkcls", "rb"))
data = Orange.data.Table("data.csv")

with the following error:
'Normalizer' object has no attribute 'need_domain'

I have been messing around changing the model and data to see where it goes wrong, but I seem to get a different 'need_domain' error with my changes so far.


Discussion:
May I ask when did you create the pickle? Was it the same version you are trying to use to open?

Also, which modeling technique are you using? Are you doing any preprocessing? Could you make a minimum workflow that builds a model like this which we can replicate, preferably using one of the documentation data sets?
Thanks for the response Marko.

May I ask when did you create the pickle? Was it the same version you are trying to use to open?  
I am not sure I exactly understand this here.  I went ahead and uninstalled Orange App this morning and removed the Python  package.  I will reinstall and rebuild the model and save to see what happens.

So I am using a few of the modeling techniques (Random Forest, Adaboost, KNN).  And yes, I am doing preprocessing.  I will work on replicating on a documentation data set -- may take me a few days to get back.

Thanks again for the response.
Got this error this morning after the reinstall "'RandomForestRegressor' object has no attribute 'min_impurity_split'"
Back to your original problem: I wanted to know if the version of Orange where you create a pickle file is the same as the one where you try to open it. I suspect they were different. Looking at the code it seems that your pickle file was created in pre-3.28.0 Orange and you were opening it in a newer version. Can you confirm this?

We usually try to take care of pickle compatibility, but sometimes we miss errors like this. I was the person who actually introduced this bug.
About the `min_impurity_split` of `RandomForestRegressor`: the same issue, but we can not do anything about this one. Normally, in Python programming, pickle files should not be treated as a stable storage, because they depend on the implementation, which can change between versions. So pickling with one version and unpickling is not guaranteed to work (but usually does so we, programmers, tend to forget about it). Sometimes it is possible to define pickling/unpickling functions manually to support pickles from different versions, but this is only done in some rare cases.

Orange internally uses scikit-learn's RF implementation, so when you save a learner, it, among other Orange properties, pickles scikit-learn's `RandomForestRegressor`. Recent scikit-learn changed RF implementation enough so that pickles are incompatible.

To solve both your problems, make sure that you use the same version of Orange and its dependencies for saving and loading models (in practice, the same versions of scikit-learn and Orange will likely suffice).
Thanks so much Marko, appreciate your help here.  Yes, the version I created this model was probably from about a year ago so that is likely the issue.

I will go ahead and close this one out and come back if issues persist after reconciling the versions.  If I just delete orange from my computer and install the latest Orange Download and then pip install the package here I would be lined up correct?

I can easily recreate the model in the new version.
I am reopening because I'd like to fix the issue. The latest Orange download and pip install would have the same versions of Orange, but dependencies could differ. Try checking which version of scikit-learn was installed by doing `pip freeze` and then updating accordingly.

Resolving patch:
diff --git a/Orange/preprocess/transformation.py b/Orange/preprocess/transformation.py
--- a/Orange/preprocess/transformation.py
+++ b/Orange/preprocess/transformation.py
@@ -16,12 +16,28 @@ def __init__(self, variable):
         :type variable: int or str or :obj:`~Orange.data.Variable`
         """
         self.variable = variable
+        self._create_cached_target_domain()
 
+    def _create_cached_target_domain(self):
+        """ If the same domain is used everytime this allows better caching of
+        domain transformations in from_table"""
         if self.variable is not None:
             if self.variable.is_primitive():
-                self.need_domain = Domain([self.variable])
+                self._target_domain = Domain([self.variable])
             else:
-                self.need_domain = Domain([], metas=[self.variable])
+                self._target_domain = Domain([], metas=[self.variable])
+
+    def __getstate__(self):
+        # Do not pickle the cached domain; rather recreate it after unpickling
+        state = self.__dict__.copy()
+        state.pop("_target_domain")
+        return state
+
+    def __setstate__(self, state):
+        # Ensure that cached target domain is created after unpickling.
+        # This solves the problem of unpickling old pickled models.
+        self.__dict__.update(state)
+        self._create_cached_target_domain()
 
     def __call__(self, data):
         """
@@ -31,7 +47,7 @@ def __call__(self, data):
         inst = isinstance(data, Instance)
         if inst:
             data = Table.from_list(data.domain, [data])
-        data = data.transform(self.need_domain)
+        data = data.transform(self._target_domain)
         if self.variable.is_primitive():
             col = data.X
         else:


Test patch:
diff --git a/Orange/tests/test_transformation.py b/Orange/tests/test_transformation.py
--- a/Orange/tests/test_transformation.py
+++ b/Orange/tests/test_transformation.py
@@ -1,3 +1,4 @@
+import pickle
 import unittest
 
 import numpy as np
@@ -43,6 +44,17 @@ def test_transform_fails(self):
         trans = Transformation(self.data.domain[2])
         self.assertRaises(NotImplementedError, trans, self.data)
 
+    def test_pickling_target_domain(self):
+        data = self.data
+        trans = self.TransformationMock(data.domain[2])
+        self.assertIn("_target_domain", trans.__dict__)
+        # _target_domain should not be pickled
+        state = trans.__getstate__()
+        self.assertNotIn("_target_domain", state)
+        # _target_domain should be recreated when unpickled
+        unpickled = pickle.loads(pickle.dumps(trans))
+        self.assertIn("_target_domain", unpickled.__dict__)
+
 
 class IdentityTest(unittest.TestCase):
     def test_identity(self):