Repository: cltk/cltk

Issue:
Cannot pickle CLTK Doc containing certain data types from Spacy
**Is your feature request related to a problem? Please describe.**
I'm always frustrated when my program crashes and I lose all of the `Doc`s I generated.

**Describe the solution you'd like**
I would like to be able to serialize (or somehow save) `Doc`s.


Discussion:
Are you familiar with storing results as a pickle file? https://realpython.com/lessons/using-pickle-module/

When I am processing many documents with the cltk, I try to run them one-by-one and save that doc to disk.

Feel free to send a code snippet of what you’re trying to do. 

Feb 18, 2024 at 12:59 by ***@***.***:

>
>
>
>
>
> Is your feature request related to a problem? Please describe.
>  I'm always frustrated when my program crashes and I lose all of the > Doc> s I generated.
>
>
>
>
> Describe the solution you'd like
>  I would like to be able to serialize (or somehow save) > Doc> s.
>
>
>
>
> —
> Reply to this email directly, > view it on GitHub <https://github.com/cltk/cltk/issues/1250>> , or > unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE36B5KOKVDK5ZK4KREGTYUJTTBAVCNFSM6AAAAABDON6UTSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE2DCMJYGEZTMNI>> .
> You are receiving this because you are subscribed to this thread.> Message ID: > <cltk/cltk/issues/1250> @> github> .> com>
>
>
>


@kylepjohnson I actually tried exactly what you suggest. Here's a snippet that illustrates my problem (let me know if you would like more of my code):

```
# downloaded files of lat_text_latin_library to corpus_files above
cltk_nlp = NLP(language="lat", suppress_banner=True)
with open(corpus_files[i], "rt", encoding="utf8") as file:
    cltk_doc = cltk_nlp.analyze(text=file.read())
    pickled_cltk_doc = pickle.dumps(cltk_doc)
    b = pickle.load(pickled_cltk_doc)
```

When I tried to do this I tried to dump `pickled_cltk_doc` directly do its own file, but I get the same error here, so the example suffices:

```
    pickled_cltk_doc = pickle.dumps(cltk_doc)
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "stringsource", line 2, in spacy.tokens.morphanalysis.MorphAnalysis.__reduce_cython__      
TypeError: self.c cannot be converted to a Python object for pickling
```

I just assumed that there was something about `Doc`s that wasn't serializable. Is this supposed to work? Am I doing something wrong?
Ahh, this is a new one. I can confirm that I get the same error (it comes from spacy): https://github.com/explosion/spaCy/issues/7644

``` python
>>> pickle.dumps(doc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "stringsource", line 2, in spacy.tokens.morphanalysis.MorphAnalysis.__reduce_cython__
TypeError: self.c cannot be converted to a Python object for pickling
```

@clemsciences have you ever encountered this with Python?

@mwitz8 if this is a deal-breaker for you, you can use the the Stanza model (which is still quite good):

``` python
>>> nlp.pipeline.processes[1]
<class 'cltk.dependency.processes.LatinSpacyProcess'>
>>> from cltk.dependency.processes import LatinStanzaProcess
>>> nlp.pipeline.processes[1] = LatinStanzaProcess
>>> nepos = "Idem cum prope ad annum octogesimum prospera pervenisset fortuna, extremis temporibus magnum in odium pervenit suorum civium, primo quod cum Demade de urbe tradenda Antipatro consenserat eiusque consilio Demosthenes cum ceteris, qui bene de re publica meriti existimabantur, populi scito in exsilium erant expulsi."
>>> doc = nlp.analyze(nepos)
>>> pickle.dumps(doc) # works ok
```

> @mwitz8 if this is a deal-breaker for you, you can use the the Stanza model (which is still quite good):

The Stanza model works for what I need. Thanks!
Thank you for bringing this to our attention. Someone will need to dig into this. Likely, the answer will be to transform native Spacy data (regarding only morphology, I think) into strings, before writing its data to the CLTK's `Doc` type. I will leave this issue open to remind the world that we need to fix this sooner than later.
I'm going to work on this issue this week.
Thank you, Clément. This is a serious bug for many users (I always write my processed Doc’s as pickles.)


Apr 22, 2024 at 14:24 by ***@***.***:

>
>
>
>
> I'm going to work on this issue this week.
>
>
>
>
> —
> Reply to this email directly, > view it on GitHub <https://github.com/cltk/cltk/issues/1250#issuecomment-2070977798>> , or > unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE36D3DRB5AQVVSHTDTHDY6V52PAVCNFSM6AAAAABDON6UTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZQHE3TONZZHA>> .
> You are receiving this because you were mentioned.> Message ID: > <cltk/cltk/issues/1250/2070977798> @> github> .> com>
>
>
>
>


My first investigation is showing that the attribute `spacy_features` of `Word` (coming from the attribute `morph` of `spacy.tokens.Token`) added in the `SpacyProcess` is not serializable. This attribute has two methods that are usable to serialize it: `to_json` and `to_dict`.

Overriding `__get_state__` and `__set_state__` may fix the problem (see https://docs.python.org/3/library/pickle.html#pickling-class-instances).
A `Token` is explicitly expected not to be serialized.
![image](https://github.com/cltk/cltk/assets/13933660/0546e81f-8631-4f2e-8f67-87fe02ac3ae5)

However, `Token.morph`, which is a [`MorphAnalysis`](https://spacy.io/api/morphology#morphanalysis), fails to be serialized. 
![image](https://github.com/cltk/cltk/assets/13933660/9a1857bd-84cc-443e-a68f-9d40658496b5)

It must be said that it has a `vocab` attribute that should not be serialized. It also has a [`to_dict`](https://spacy.io/api/morphology#morphanalysis-to_dict) method that helps serialize, but I'm not sure how to deserialize it.

Should I open a new issue on https://github.com/explosion/spaCy?

What if you try `del token.morph` or `setattr()`? We need this info while writing spacy’s data to our Doc, however once we have it we do not need to save that info anymore.

You could ask the project but it’s so big you might not get a response. It’s definitely a bug, I once saw an Issue about it. 

Apr 25, 2024 at 14:20 by ***@***.***:

>
>
>
>
>
> A > Token>  is explicitly expected not to be serialized.
>  > image.png (view on web) <https://github.com/cltk/cltk/assets/13933660/0546e81f-8631-4f2e-8f67-87fe02ac3ae5>
>
>
>
>
> However, > Token.morph> , which is a > MorphAnalysis <https://spacy.io/api/morphology#morphanalysis>> , fails to be serialized.
>  > image.png (view on web) <https://github.com/cltk/cltk/assets/13933660/9a1857bd-84cc-443e-a68f-9d40658496b5>
>
>
>
> It must be said that it has a > vocab>  attribute that should not be serialized. It also has a > to_dict <https://spacy.io/api/morphology#morphanalysis-to_dict>>  method that helps serialize, but I'm not sure how to deserialize it.
>
>
> Should I open a new issue on > https://github.com/explosion/spaCy> ?
>
>
>
> —
> Reply to this email directly, > view it on GitHub <https://github.com/cltk/cltk/issues/1250#issuecomment-2078195153>> , or > unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE36BJPD3E6EAQ4I47ZJDY7FXSDAVCNFSM6AAAAABDON6UTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZYGE4TKMJVGM>> .
> You are receiving this because you were mentioned.> Message ID: > <cltk/cltk/issues/1250/2078195153> @> github> .> com>
>
>
>
In the `SpacyProcess`, 

```python
    def spacy_to_cltk_word_type(spacy_doc: spacy.tokens.doc.Doc):
        """...
        """
        words_list: list[Word] = []
        for sentence_index, sentence in enumerate(spacy_doc.doc.sents):
            sent_words: dict[int, Word] = {}
            for spacy_word in sentence:
                pos: Optional[MorphosyntacticFeature] = None
                if spacy_word.pos_:
                    pos = from_ud("POS", spacy_word.pos_)
                cltk_word = Word(
                    # Note: In order to match how Stanza orders token output
                    # (index starting at 1, not 0), we must add an extra 1 to each
                    index_token=spacy_word.i + 1,
                    index_char_start=spacy_word.idx,
                    index_char_stop=spacy_word.idx + len(spacy_word),
                    index_sentence=sentence_index,
                    string=spacy_word.text,  # same as ``token.text``
                    pos=pos,
                    xpos=spacy_word.tag_,
                    upos=spacy_word.pos_,
                    lemma=spacy_word.lemma_,
                    dependency_relation=spacy_word.dep_,  # str
                    stop=spacy_word.is_stop,
                    # Note: Must increment this, too
                    governor=spacy_word.head.i + 1,  # TODO: Confirm this is the index
                )
                raw_features: list[tuple[str, str]] = (
                    [
                        (feature, value)
                        for feature, value in spacy_word.morph.to_dict().items()
                    ]
                    if spacy_word.morph
                    else []
                )
                cltk_features = [
                    from_ud(feature_name, feature_value)
                    for feature_name, feature_value in raw_features
                ]
                cltk_word.features = MorphosyntacticFeatureBundle(*cltk_features)
                cltk_word.category = to_categorial(cltk_word.pos)
                cltk_word.spacy_features = spacy_word.morph
                sent_words[cltk_word.index_token] = cltk_word
                words_list.append(cltk_word)
        return words_list
```

`cltk_word.features` already has all the info contained in `spacy_word.morph`. I suggest that we remove `cltk_word.spacy_features` because it's redundant with `cltk_word.features`.

Resolving patch:
diff --git a/src/cltk/dependency/processes.py b/src/cltk/dependency/processes.py
--- a/src/cltk/dependency/processes.py
+++ b/src/cltk/dependency/processes.py
@@ -320,7 +320,6 @@ def spacy_to_cltk_word_type(spacy_doc: spacy.tokens.doc.Doc):
                 ]
                 cltk_word.features = MorphosyntacticFeatureBundle(*cltk_features)
                 cltk_word.category = to_categorial(cltk_word.pos)
-                cltk_word.spacy_features = spacy_word.morph
                 sent_words[cltk_word.index_token] = cltk_word
                 words_list.append(cltk_word)
         return words_list


Test patch: