Repository: bab2min/tomotopy (language: python)

Issue:
Calling `infer` causes a crash
Hi I need to extract the documents per inferred topic using HDP, how can I do this? Thank you.


Discussion / hints:
Hello @RUMCS, sorry for late feedback.
What you want is to pick documents based on its topic, isn't it?
Since, in topic models including HDP, every document contains many topics, it is ambiguous to decide which document is on a particular topic.
Let's briefly define 'relatedness' to certain topics of a document as follows:
- A document is related to topic X if topic X is in the top 'n' among the distribution of topics in the document.

In this case, the documents related to each topic can be extracted with following codes.

```
# we use only top-1 topic for relatedness
top_n = 1

# initialize model with appropriate parameters
mdl = tp.HDPModel()

#
# ... add_doc and training ...
#

# prepare documents for inference
docs = []
for line in open('some_docs.txt'):
  docs.append(mdl.make_doc(line.strip().split()))

# let's infer
dist, ll = mdl.infer(docs)

# extract documents related to each topic
docs_per_topic = [[] for _ in range(mdl.k)]
for doc in docs:
  for topic, weight in doc.get_topics(top_n=top_n):
    docs_per_topic[topic].append(doc)

# print documents related to each topic
for k in range(mdl.k):
  if not mdl.is_live_topic(k): continue
  print('Topic #{} {}'.format(k, ' '.join(word for word, _ in mdl.get_topic_words(k, top_n=5))))
  for doc in docs_per_topic[k]:
    print(doc)
  print()
```

You can control the extracted documents by chaning top_n value or adding another condition such as weight threshold of topics.
Hello @bab2min 
the last part of the code didn't work for me.  May you please help.

TypeError: function missing required argument 'topic_id' (pos 1) for line starts with print('Topic #{} {}'.format(k, etc

SyntaxError: invalid syntax for doc in docs_per_topic[k]:

![error](https://user-images.githubusercontent.com/46090628/90965075-e935b700-e4d5-11ea-80a6-c8b63d776622.jpg)

can I get the output in the following format:
Doc No.1 : Topic #3 
Doc No.2 : Topic #10 
Doc No.3 : Topic #3 
Doc No.4 : Topic #3 
etc
![Format](https://user-images.githubusercontent.com/46090628/90965324-2307bd00-e4d8-11ea-893f-9a733861bc07.jpg)

@fuaduae Oops, sorry for late answer.
The code in the above comment has typo errors.

`mdl.get_topic_words(top_n=5)` should be corrected to `mdl.get_topic_words(k, top_n=5)`.

```
for k in range(mdl.k):
  if not mdl.is_live_topic(k): continue
  print('Topic #{} {}'.format(k, ' '.join(word for word, _ in mdl.get_topic_words(k, top_n=5))))
  for doc in docs_per_topic[k]:
    print(doc)
  print()
```

I am trying to get topic probabilities for each document after training an HDP model but the kernel keeps dying. I only have 12,000 documents and I tried scoring in 100 document chunks but still the same issue. I tried changing together parameter to False but it didn't make any difference either. 
I was able to generate topic probabilities on the same set of documents with no issues using  LDA model.

`for i in range(0, len(texts), 100):
doc_inst = [model.make_doc(text) for text in texts[i:i+100]]
review_topics = model.infer(doc_inst, together = True)`



Any suggestions?
Hi @Jurgita-DS,
Thank you for your reporting.
Could you share the version of your python and tomotopy where the problem occurs?
Hi @bab2min,
I am using python version 3.7.4 and tomotopy  0.10.2.

The issue seems to be somehow related with scoring certain document instances, however, I cannot see anything wrong with those documents.
texts is a list of documents where each document is a list of tokens, e.g. [[word1, word2], [word3, word4, word5]].
If I, for example, exclude the 8000th document the below code works fine.

```
doc_inst = [model.make_doc(text) for text in texts[8001:8500]]
doc_topics = model.infer(doc_inst, together = True)
```
Also if I only include the 8000th document the code runs with no issue.

```
doc_inst = [model.make_doc(text) for text in texts[8000]]
doc_topics = model.infer(doc_inst, together = True)
```

However, neither of the below two options work. Both cause the kernel to die.

```
# Generate topics for documents 8000 and 8001st
doc_inst = [model.make_doc(text) for text in texts[8000:8002]]
doc_topics = model.infer(doc_inst, together = True)

# Generate topics for document 8000th
doc_inst = [model.make_doc(text) for text in texts[8000:8001]]
doc_topics = model.infer(doc_inst, together = True)
```
I cannot see anything wrong with the document 8000. It contains a small number of words which all look fine to me.

I have also tried converting the HDP model into LDA, however, got the below error.
'there is a document with uid =  already'
This is the code I used:
`model_conv = model.convert_to_lda(topic_threshold = 0)`

I am not sure if it is related with the first issue of kernel dying or it's a separate one.
I have checked that there are no empty documents in the corpus (all contain at least one token), and no strange tokens (e.g. space).

I hope this helps. 
I am working with the data that I cannot be shared publicly so I am not able to share the data and the model that is causing the issue. I tried running the same code on the open source data but wasn't able to reproduce the error.
@Jurgita-DS
Thank you for sharing details.
It seems that the combination of the 8000th document's content and the trained HDPModel cause a crash.
Probably there is a potential flaw in the code on the current `infer` method of HDPModel. As a result of writing the test code as follows in the same environment as the one you shared with, we found that a crash occurs in the infer function with some probability.

Based on your situation , I wrote the following test code.
When I ran it in the same environment that you shared with me, I confirmed that a crash occurs in the infer method sometimes.
```
while 1:
  import tomotopy as tp
  mdl = tp.HDPModel(min_cf=2)
  mdl.add_doc("a b c d e f".split())
  mdl.add_doc("1 2 3 d e f".split())
  mdl.add_doc("c d".split())
  mdl.add_doc("3 f".split())
  
  mdl.train(1000)
  
  doc1 = mdl.make_doc("a b c".split())
  doc2 = mdl.make_doc("b c d e".split())
  
  mdl.infer([doc1, doc2])
```

Also, the problem where `convert_to_lda` raises `there is a document with uid = already` exception seems another bug. 

I will try to determine the cause of two bugs you reported and fix them.
Thank you for your reporting again!

Fix patch:
diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
index b6382ce..a03e4a0 100644
--- a/.github/workflows/deploy.yml
+++ b/.github/workflows/deploy.yml
@@ -40,12 +40,12 @@ jobs:
         TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
         TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
       run: |
-        /opt/python/cp38-cp38/bin/python -m pip install twine wheel numpy
+        /opt/python/cp38-cp38/bin/python -m pip install twine wheel numpy==`/opt/python/cp38-cp38/bin/python test/numpy_version.py`
         /opt/python/cp38-cp38/bin/python setup.py sdist
         /opt/python/cp38-cp38/bin/python -m twine upload dist/*.tar.gz
         for cp in cp35-cp35m cp36-cp36m cp37-cp37m cp38-cp38 cp39-cp39
         do
-          /opt/python/${cp}/bin/python -m pip install twine wheel numpy
+          /opt/python/${cp}/bin/python -m pip install twine wheel numpy==`/opt/python/${cp}/bin/python test/numpy_version.py`
           /opt/python/${cp}/bin/python setup.py build bdist_wheel
           auditwheel repair dist/*-${cp}-linux_x86_64.whl
         done
@@ -84,7 +84,7 @@ jobs:
         TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
       run: |
         export AUDITWHEEL_PLAT=many
-        python -m pip install twine wheel numpy
+        python -m pip install twine wheel numpy==`python test/numpy_version.py`
         python setup.py bdist_wheel
         twine upload dist/*
 
@@ -122,6 +122,6 @@ jobs:
         TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
         TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
       run: |
-        python -m pip install twine wheel numpy
+        python -m pip install twine wheel numpy==$(python test/numpy_version.py)
         python setup.py bdist_wheel
         twine upload dist/*
diff --git a/.github/workflows/generate_documentation.yml b/.github/workflows/generate_documentation.yml
index 7c55f80..9ba05ee 100644
--- a/.github/workflows/generate_documentation.yml
+++ b/.github/workflows/generate_documentation.yml
@@ -39,7 +39,7 @@ jobs:
         mv variant/include/mapbox include/
     - name: build
       run: |
-        python3 -m pip install numpy
+        python3 -m pip install numpy==1.10.4
         python3 -m pip install pdoc3==0.8.4
         export TOMOTOPY_LANG=${{ matrix.language }}
         python3 setup.py install
diff --git a/setup.py b/setup.py
index d68405a..d6127f3 100644
--- a/setup.py
+++ b/setup.py
@@ -93,10 +93,10 @@
         "Operating System :: POSIX",
         "Operating System :: MacOS"
     ],
-    install_requires=['numpy>=1.10.0'],
+    install_requires=['numpy>=1.11.0'],
     keywords='NLP,Topic Model',
 
-    packages = ['tomotopy'],
+    packages=['tomotopy'],
     include_package_data=True,
-    ext_modules = modules
+    ext_modules=modules
 )
diff --git a/src/TopicModel/HDPModel.hpp b/src/TopicModel/HDPModel.hpp
index 65935c3..41c5461 100644
--- a/src/TopicModel/HDPModel.hpp
+++ b/src/TopicModel/HDPModel.hpp
@@ -403,52 +403,40 @@ namespace tomoto
 			if (_tw != TermWeight::one) doc.wordWeights.resize(wordSize);
 		}
 
-		template<bool _Infer>
+		template<bool _infer>
 		void updateStateWithDoc(typename BaseClass::Generator& g, _ModelState& ld, _RandGen& rgs, _DocType& doc, size_t i) const
 		{
-			// generate tables for each topic when inferring
-			if (_Infer)
+			Tid t;
+			std::vector<double> dist;
+			dist.emplace_back(this->alpha);
+			for (auto& d : doc.numTopicByTable) dist.emplace_back(d.num);
+			std::discrete_distribution<Tid> ddist{ dist.begin(), dist.end() };
+			t = ddist(rgs);
+			if (t == 0)
 			{
-				if (i < this->K)
+				// new table
+				Tid k;
+				if (_infer)
 				{
-					Tid t = i;
-					if (isLiveTopic(i))
+					std::uniform_int_distribution<> theta{ 0, this->K - 1 };
+					do
 					{
-						t = doc.addNewTable(i);
-					}
-					else
-					{
-						t = std::uniform_int_distribution<size_t>{ 0, doc.getNumTable() - 1 }(rgs);
-					}
-					++ld.numTableByTopic[doc.numTopicByTable[t].topic];
-					++ld.totalTable;
-					doc.Zs[i] = t;
-				}
-				else doc.Zs[i] = std::uniform_int_distribution<size_t>{ 0, doc.getNumTable() - 1 }(rgs);
-			}
-			// generate tables following CRP
-			else
-			{
-				Tid t;
-				std::vector<double> dist;
-				dist.emplace_back(this->alpha);
-				for (auto& d : doc.numTopicByTable) dist.emplace_back(d.num);
-				std::discrete_distribution<Tid> ddist{ dist.begin(), dist.end() };
-				t = ddist(rgs);
-				if (t == 0)
-				{
-					// new table
-					Tid k = g.theta(rgs);
-					t = doc.addNewTable(k);
-					++ld.numTableByTopic[k];
-					++ld.totalTable;
+						k = theta(rgs);
+					} while (!isLiveTopic(k));
 				}
 				else
 				{
-					t -= 1;
+					k = g.theta(rgs);
 				}
-				doc.Zs[i] = t;
+				t = doc.addNewTable(k);
+				++ld.numTableByTopic[k];
+				++ld.totalTable;
+			}
+			else
+			{
+				t -= 1;
 			}
+			doc.Zs[i] = t;
 			addWordTo<1>(ld, doc, i, doc.words[i], doc.Zs[i], doc.numTopicByTable[doc.Zs[i]].topic);
 		}
 
diff --git a/src/TopicModel/TopicModel.hpp b/src/TopicModel/TopicModel.hpp
index 719e3cd..bc26ee0 100644
--- a/src/TopicModel/TopicModel.hpp
+++ b/src/TopicModel/TopicModel.hpp
@@ -373,7 +373,7 @@ namespace tomoto
 		{
 			if (doc.words.empty()) return -1;
 			if (!doc.docUid.empty() && uidMap.count(doc.docUid))
-				throw exception::InvalidArgument{ "there is a document with uid = " + std::string{ doc.docUid } + " already." };
+				throw exception::InvalidArgument{ "there is a document with uid = '" + std::string{ doc.docUid } + "' already." };
 			size_t maxWid = *std::max_element(doc.words.begin(), doc.words.end());
 			if (vocabCf.size() <= maxWid)
 			{
@@ -383,7 +383,7 @@ namespace tomoto
 			for (auto w : doc.words) ++vocabCf[w];
 			std::unordered_set<Vid> uniq{ doc.words.begin(), doc.words.end() };
 			for (auto w : uniq) ++vocabDf[w];
-			uidMap.emplace(doc.docUid, docs.size());
+			if (!doc.docUid.empty()) uidMap.emplace(doc.docUid, docs.size());
 			docs.emplace_back(std::forward<_DocTy>(doc));
 			return docs.size() - 1;
 		}
diff --git a/src/Utils/SharedString.hpp b/src/Utils/SharedString.hpp
index b9fcdbb..77a060b 100644
--- a/src/Utils/SharedString.hpp
+++ b/src/Utils/SharedString.hpp
@@ -109,7 +109,7 @@ namespace tomoto
 
 		bool empty() const
 		{
-			return ptr == nullptr;
+			return ptr == nullptr || size() == 0;
 		}
 
 		operator std::string() const


Test patch:
diff --git a/.github/workflows/deploy_test.yml b/.github/workflows/deploy_test.yml
index 9f4a7ac..cfa4e47 100644
--- a/.github/workflows/deploy_test.yml
+++ b/.github/workflows/deploy_test.yml
@@ -39,12 +39,12 @@ jobs:
         TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
         TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
       run: |
-        /opt/python/cp35-cp35m/bin/python -m pip install twine wheel numpy
+        /opt/python/cp35-cp35m/bin/python -m pip install twine wheel numpy==`/opt/python/cp35-cp35m/bin/python test/numpy_version.py`
         /opt/python/cp35-cp35m/bin/python setup.py sdist
         /opt/python/cp35-cp35m/bin/python -m twine upload --repository testpypi dist/*.tar.gz
         for cp in cp35-cp35m cp36-cp36m cp37-cp37m cp38-cp38
         do
-          /opt/python/${cp}/bin/python -m pip install twine wheel numpy
+          /opt/python/${cp}/bin/python -m pip install twine wheel numpy==`/opt/python/${cp}/bin/python test/numpy_version.py`
           /opt/python/${cp}/bin/python setup.py build bdist_wheel
           auditwheel repair dist/*-${cp}-linux_x86_64.whl
         done
@@ -83,7 +83,7 @@ jobs:
         TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
       run: |
         export AUDITWHEEL_PLAT=many
-        python -m pip install twine wheel numpy
+        python -m pip install twine wheel numpy==`python test/numpy_version.py`
         python setup.py bdist_wheel
         twine upload --repository testpypi dist/*
 
@@ -121,6 +121,6 @@ jobs:
         TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
         TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
       run: |
-        python -m pip install twine wheel numpy
+        python -m pip install twine wheel numpy==$(python test/numpy_version.py)
         python setup.py bdist_wheel
         twine upload --repository testpypi dist/*
diff --git a/.github/workflows/pull_request_test.yml b/.github/workflows/pull_request_test.yml
index b7c5f2d..ad68dd0 100644
--- a/.github/workflows/pull_request_test.yml
+++ b/.github/workflows/pull_request_test.yml
@@ -40,7 +40,7 @@ jobs:
         mv variant/include/mapbox include/
     - name: Build
       run: |
-        /opt/python/${{ matrix.cp }}/bin/python -m pip install numpy
+        /opt/python/${{ matrix.cp }}/bin/python -m pip install numpy==`/opt/python/${{ matrix.cp }}/bin/python test/numpy_version.py`
         /opt/python/${{ matrix.cp }}/bin/python setup.py build install
     - name: Test
       run: |
@@ -76,7 +76,7 @@ jobs:
         mv variant-1.1.3/include/mapbox include/
     - name: Build
       run: |
-        python -m pip install numpy
+        python -m pip install numpy==`python test/numpy_version.py`
         python setup.py build install
     - name: Test
       run: |
@@ -115,7 +115,7 @@ jobs:
         mv variant-1.1.3/include/mapbox include/
     - name: Build
       run: |
-        python -m pip install numpy
+        python -m pip install numpy==$(python test/numpy_version.py)
         python setup.py build install
     - name: Archive binary
       uses: actions/upload-artifact@v2
diff --git a/test/numpy_version.py b/test/numpy_version.py
new file mode 100644
index 0000000..71b4f3d
--- /dev/null
+++ b/test/numpy_version.py
@@ -0,0 +1,14 @@
+
+def get_old_numpy_version():
+    import sys
+
+    py_version = sys.version_info
+    if py_version >= (3, 9): return '1.20.0'
+    if py_version >= (3, 8): return '1.18.0'
+    if py_version >= (3, 7): return '1.15.0'
+    if py_version >= (3, 6): return '1.12.0'
+    if py_version >= (3, 5): return '1.11.0'
+    return '1.10.0'
+
+if __name__ == '__main__':
+    print(get_old_numpy_version())