Repository: andrewtavis/kwx (language: python)

Issue:
Create concise requirement and env files
This issue is for creating concise versions of [requirements.txt](https://github.com/andrewtavis/kwx/blob/main/requirements.txt) and [environment.yml](https://github.com/andrewtavis/kwx/blob/main/environment.yml) for kwx. It would be great if these files were created by hand with specific version numbers or generated in a way so that sub-dependencies don't always need to be updated. 

As of now both files are being created with the following commands in the package's conda virtual environment:
```bash
pip list --format=freeze > requirements.txt  
conda env export --no-builds | grep -v "^prefix: " > environment.yml
```

kwx, en-core-web-sm (spacy package that breaks tests), and other obviously unneeded packages are then removed from these files before being uploaded. 

Any insights or help would be much appreciated!


Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index a095a20..8bb7ba8 100644
--- a/README.md
+++ b/README.md
@@ -232,8 +232,6 @@ gen_word_cloud(
 
 Please see the [contribution guidelines](https://github.com/andrewtavis/kwx/blob/main/.github/CONTRIBUTING.md) if you are interested in contributing to this project. Work that is in progress or could be implemented includes:
 
-- Creating concise [requirements.txt](https://github.com/andrewtavis/kwx/blob/main/requirements.txt) and [environment.yml](https://github.com/andrewtavis/kwx/blob/main/environment.yml) files [(see issue)](https://github.com/andrewtavis/kwx/issues/23)
-
 - Including more methods to extract keywords [(see issue)](https://github.com/andrewtavis/kwx/issues/17)
 
 - Adding key phrase extraction as an option for [kwx.model.extract_kws](https://github.com/andrewtavis/kwx/blob/main/src/kwx/model.py) [(see issues)](https://github.com/andrewtavis/kwx/issues/)
diff --git a/docs/source/conf.py b/docs/source/conf.py
index 2560d77..c9c5a70 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -68,6 +68,7 @@
     "IPython",
     "sentence_transformers",
     "tqdm",
+    "defusedxml",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
diff --git a/environment.yml b/environment.yml
index 4304886..e53210a 100644
--- a/environment.yml
+++ b/environment.yml
@@ -4,262 +4,29 @@ channels:
   - anaconda
   - defaults
 dependencies:
-  - _tflow_select=2.3.0
-  - absl-py=0.11.0
-  - aiohttp=3.7.4
-  - appdirs=1.4.4
-  - appnope=0.1.2
-  - argon2-cffi=20.1.0
-  - astor=0.8.1
-  - astroid=2.5
-  - async-timeout=3.0.1
-  - async_generator=1.10
-  - attrs=20.3.0
-  - backcall=0.2.0
-  - blas=1.0
-  - bleach=3.3.0
-  - boto=2.49.0
-  - boto3=1.9.66
-  - botocore=1.12.67
-  - bz2file=0.98
-  - c-ares=1.17.1
-  - ca-certificates=2021.1.19
-  - cachetools=4.2.1
-  - catalogue=1.0.0
+  - black=20.8b1
   - certifi=2020.12.5
-  - cffi=1.14.4
-  - chardet=3.0.4
-  - click=7.1.2
-  - cmarkgfm=0.4.2
-  - colorama=0.4.4
-  - coverage=5.4
-  - cryptography=3.4.4
-  - cycler=0.10.0
-  - cymem=2.0.4
-  - cython-blis=0.4.1
-  - decorator=4.4.2
   - defusedxml=0.7.1
-  - docutils=0.16
-  - freetype=2.10.4
-  - future=0.18.2
   - gensim=3.8.0
-  - google-api-core=1.22.2
-  - google-auth=1.24.0
-  - google-cloud-core=1.5.0
-  - google-cloud-storage=1.35.0
-  - google-crc32c=1.1.1
-  - google-resumable-media=1.2.0
-  - googleapis-common-protos=1.52.0
-  - h5py=2.10.0
-  - hdf5=1.10.6
-  - idna=2.10
-  - importlib-metadata=2.0.0
-  - importlib_metadata=2.0.0
-  - iniconfig=1.1.1
-  - intel-openmp=2019.4
-  - ipython_genutils=0.2.0
-  - ipywidgets=7.6.3
-  - isort=5.8.0
-  - jedi=0.17.2
-  - jinja2=2.11.3
-  - jmespath=0.10.0
-  - joblib=1.0.1
-  - jpeg=9b
-  - jsonschema=3.0.2
-  - jupyter_client=6.1.7
-  - jupyter_core=4.7.1
-  - jupyterlab_pygments=0.1.2
-  - jupyterlab_widgets=1.0.0
   - keras=2.3.1
-  - keras-applications=1.0.8
-  - keras-base=2.3.1
-  - keras-preprocessing=1.1.2
-  - keyring=22.0.1
-  - kiwisolver=1.3.1
-  - lazy-object-proxy=1.6.0
-  - lcms2=2.11
-  - libcrc32c=1.1.1
-  - libcxx=10.0.0
-  - libedit=3.1.20191231
-  - libffi=3.3
-  - libgfortran=3.0.1
-  - libpng=1.6.37
-  - libprotobuf=3.14.0
-  - libsodium=1.0.18
-  - libtiff=4.1.0
-  - llvm-openmp=10.0.0
-  - lz4-c=1.9.3
-  - markdown=3.3.3
-  - markupsafe=1.1.1
   - matplotlib=3.3.2
-  - matplotlib-base=3.3.2
-  - mccabe=0.6.1
-  - mistune=0.8.4
-  - mkl=2019.4
-  - mkl-service=2.3.0
-  - mkl_fft=1.2.0
-  - mkl_random=1.1.1
-  - more-itertools=8.6.0
-  - multidict=4.7.6
-  - murmurhash=1.0.5
-  - mypy_extensions=0.4.3
-  - nb_conda_kernels=2.3.1
-  - nbconvert=6.0.7
-  - nbformat=5.1.2
-  - ncurses=6.2
-  - nest-asyncio=1.5.1
   - nltk=3.5
-  - notebook=6.2.0
   - numpy=1.19.2
-  - numpy-base=1.19.2
-  - olefile=0.46
-  - openssl=1.1.1k
   - packaging=20.9
   - pandas=1.2.1
-  - pandoc=2.12
-  - pandocfilters=1.4.3
-  - parso=0.7.0
-  - pathspec=0.7.0
-  - patsy=0.5.1
-  - pexpect=4.8.0
-  - pickleshare=0.7.5
-  - pillow=8.1.2
-  - pip=20.3.3
-  - pkginfo=1.7.0
-  - plac=1.1.0
-  - pluggy=0.13.1
-  - preshed=3.0.2
-  - prometheus_client=0.9.0
-  - prompt-toolkit=3.0.8
-  - protobuf=3.14.0
-  - ptyprocess=0.7.0
-  - py=1.10.0
-  - pyasn1=0.4.8
-  - pyasn1-modules=0.2.8
-  - pycparser=2.20
-  - pygments=2.7.4
-  - pylint=2.6.0
-  - pyopenssl=20.0.1
-  - pyparsing=2.4.7
-  - pyrsistent=0.17.3
-  - pysocks=1.7.1
   - pytest=6.2.2
   - pytest-cov=2.11.1
-  - python=3.7.9
-  - python-dateutil=2.8.1
-  - python_abi=3.7
-  - pytz=2021.1
-  - pyyaml=5.4.1
-  - readline=8.1
-  - readme_renderer=24.0
-  - regex=2020.11.13
-  - requests=2.25.1
-  - requests-toolbelt=0.9.1
-  - rfc3986=1.4.0
-  - rope=0.18.0
-  - rsa=4.7
-  - s3transfer=0.1.13
   - scikit-learn=0.23.2
-  - scipy=1.6.0
   - seaborn=0.11.1
-  - seaborn-base=0.11.1
-  - send2trash=1.5.0
-  - setuptools=52.0.0
-  - six=1.15.0
-  - smart_open=4.1.2
   - spacy=2.3.5
-  - sqlite=3.33.0
-  - srsly=1.0.5
-  - statsmodels=0.12.2
-  - tensorflow-base=1.14.0
-  - termcolor=1.1.0
-  - testpath=0.4.4
-  - thinc=7.4.5
-  - threadpoolctl=2.1.0
-  - tk=8.6.10
-  - toml=0.10.1
-  - tornado=6.1
   - tqdm=4.56.0
-  - traitlets=5.0.5
-  - twine=3.3.0
-  - typed-ast=1.4.2
-  - typing-extensions=3.7.4.3
-  - typing_extensions=3.7.4.3
-  - urllib3=1.24.3
-  - wasabi=0.8.2
-  - wcwidth=0.2.5
-  - webencodings=0.5.1
-  - werkzeug=1.0.1
-  - wheel=0.36.2
-  - widgetsnbextension=3.5.1
   - wordcloud=1.8.1
-  - wrapt=1.12.1
   - xlrd=2.0.1
-  - xz=5.2.5
-  - yaml=0.2.5
-  - yarl=1.5.1
-  - zeromq=4.3.4
-  - zipp=3.4.0
-  - zlib=1.2.11
-  - zstd=1.4.5
   - pip:
-    - alabaster==0.7.12
-    - astunparse==1.6.3
-    - babel==2.9.0
-    - black==20.8b1
     - emoji==1.2.0
-    - entrypoints==0.3
-    - filelock==3.0.12
-    - flatbuffers==1.12
-    - funcy==1.15
-    - gast==0.3.3
-    - google-auth-oauthlib==0.4.3
-    - google-pasta==0.2.0
     - googletrans==4.0.0rc1
-    - grpcio==1.32.0
-    - h11==0.9.0
-    - h2==3.2.0
-    - hpack==3.0.0
-    - hstspreload==2020.12.22
-    - httpcore==0.9.1
-    - httpx==0.13.3
-    - hyperframe==5.2.0
-    - imagesize==1.2.0
-    - ipykernel==5.4.3
     - ipython==7.10.0
-    - jupyter-client==6.1.11
-    - jupyter-contrib-core==0.3.3
-    - jupyter-contrib-nbextensions==0.5.1
-    - jupyter-highlight-selected-word==0.2.0
-    - jupyter-latex-envs==1.4.6
-    - jupyter-nbextensions-configurator==0.4.1
-    - lxml==4.6.3
-    - nbclient==0.5.2
-    - numexpr==2.7.2
-    - oauthlib==3.1.0
-    - opt-einsum==3.3.0
     - pyldavis==3.2.0
-    - pyzmq==22.0.3
-    - requests-oauthlib==1.3.0
-    - sacremoses==0.0.43
     - sentence-transformers==0.4.1.2
-    - sentencepiece==0.1.95
-    - sniffio==1.2.0
-    - snowballstemmer==2.1.0
-    - sphinx==3.5.1
-    - sphinx-rtd-theme==0.5.1
-    - sphinxcontrib-applehelp==1.0.2
-    - sphinxcontrib-devhelp==1.0.2
-    - sphinxcontrib-htmlhelp==1.0.3
-    - sphinxcontrib-jsmath==1.0.1
-    - sphinxcontrib-qthelp==1.0.3
-    - sphinxcontrib-serializinghtml==1.1.4
     - stopwordsiso==0.6.1
-    - tensorboard==2.4.1
-    - tensorboard-plugin-wit==1.8.0
     - tensorflow==2.4.1
-    - tensorflow-estimator==2.4.0
-    - terminado==0.9.2
-    - tokenizers==0.10.1
-    - torch==1.7.1
-    - transformers==4.3.2
diff --git a/requirements.txt b/requirements.txt
index 874e5ee..c54d941 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,216 +1,25 @@
-absl-py==0.11.0
-aiohttp==3.7.4
-alabaster==0.7.12
-appdirs==1.4.4
-appnope==0.1.2
-argon2-cffi==20.1.0
-astor==0.8.1
-astroid==2.5
-astunparse==1.6.3
-async-generator==1.10
-async-timeout==3.0.1
-attrs==20.3.0
-Babel==2.9.0
-backcall==0.2.0
 black==20.8b1
-bleach==3.3.0
-blis==0.4.1
-boto==2.49.0
-boto3==1.9.66
-botocore==1.12.67
-bz2file==0.98
-cachetools==4.2.1
-catalogue==1.0.0
 certifi==2020.12.5
-cffi==1.14.4
-chardet==3.0.4
-click==7.1.2
-cmarkgfm==0.4.2
-colorama==0.4.4
-coverage==5.4
-cryptography==3.4.4
-cycler==0.10.0
-cymem==2.0.4
-decorator==4.4.2
 defusedxml==0.7.1
-docutils==0.16
 emoji==1.2.0
-entrypoints==0.3
-filelock==3.0.12
-flatbuffers==1.12
-funcy==1.15
-future==0.18.2
-gast==0.4.0
 gensim==3.8.0
-google-api-core==1.22.2
-google-auth==1.24.0
-google-auth-oauthlib==0.4.3
-google-cloud-core==1.5.0
-google-cloud-storage==1.35.0
-google-crc32c==1.1.1
-google-pasta==0.2.0
-google-resumable-media==1.2.0
-googleapis-common-protos==1.52.0
 googletrans==4.0.0rc1
-grpcio==1.32.0
-h11==0.9.0
-h2==3.2.0
-h5py==2.10.0
-hpack==3.0.0
-hstspreload==2020.12.22
-httpcore==0.9.1
-httpx==0.13.3
-hyperframe==5.2.0
-idna==2.10
-imagesize==1.2.0
-importlib-metadata==2.0.0
-iniconfig==1.1.1
-ipykernel==5.4.3
 ipython==7.10.0
-ipython-genutils==0.2.0
-ipywidgets==7.6.3
-isort==5.8.0
-jedi==0.17.2
-Jinja2==2.11.3
-jmespath==0.10.0
-joblib==1.0.1
-jsonschema==3.0.2
-jupyter-client==6.1.11
-jupyter-contrib-core==0.3.3
-jupyter-contrib-nbextensions==0.5.1
-jupyter-core==4.7.1
-jupyter-highlight-selected-word==0.2.0
-jupyter-latex-envs==1.4.6
-jupyter-nbextensions-configurator==0.4.1
-jupyterlab-pygments==0.1.2
-jupyterlab-widgets==1.0.0
-Keras==2.3.1
-Keras-Applications==1.0.8
-Keras-Preprocessing==1.1.2
-keyring==22.0.1
-kiwisolver==1.3.1
-lazy-object-proxy==1.6.0
-lxml==4.6.3
-Markdown==3.3.3
-MarkupSafe==1.1.1
+keras==2.3.1
 matplotlib==3.3.2
-mccabe==0.6.1
-mistune==0.8.4
-mkl-fft==1.2.0
-mkl-random==1.1.1
-mkl-service==2.3.0
-more-itertools==8.6.0
-multidict==4.7.6
-murmurhash==1.0.5
-mypy-extensions==0.4.3
-nb-conda-kernels==2.3.1
-nbclient==0.5.3
-nbconvert==6.0.7
-nbformat==5.1.2
-nest-asyncio==1.5.1
 nltk==3.5
-notebook==6.2.0
-numexpr==2.7.2
 numpy==1.19.2
-oauthlib==3.1.0
-olefile==0.46
-opt-einsum==3.3.0
 packaging==20.9
 pandas==1.2.1
-pandocfilters==1.4.3
-parso==0.7.0
-pathspec==0.7.0
-patsy==0.5.1
-pexpect==4.8.0
-pickleshare==0.7.5
-Pillow==8.1.2
-pip==20.3.3
-pkginfo==1.7.0
-plac==1.1.0
-pluggy==0.13.1
-preshed==3.0.2
-prometheus-client==0.9.0
-prompt-toolkit==3.0.8
-protobuf==3.14.0
-ptyprocess==0.7.0
-py==1.10.0
-pyasn1==0.4.8
-pyasn1-modules==0.2.8
-pycparser==2.20
-Pygments==2.7.4
-pyLDAvis==3.2.0
-pylint==2.6.0
-pyOpenSSL==20.0.1
-pyparsing==2.4.7
-pyrsistent==0.17.3
-PySocks==1.7.1
+pyldavis==3.2.0
 pytest==6.2.2
 pytest-cov==2.11.1
-python-dateutil==2.8.1
-pytz==2021.1
-PyYAML==5.4.1
-pyzmq==22.0.3
-readme-renderer==28.0
-regex==2020.11.13
-requests==2.25.1
-requests-oauthlib==1.3.0
-requests-toolbelt==0.9.1
-rfc3986==1.4.0
-rope==0.18.0
-rsa==4.7
-s3transfer==0.1.13
-sacremoses==0.0.43
 scikit-learn==0.23.2
-scipy==1.6.0
 seaborn==0.11.1
-Send2Trash==1.5.0
 sentence-transformers==0.4.1.2
-sentencepiece==0.1.95
-setuptools==52.0.0.post20210125
-six==1.15.0
-smart-open==4.1.2
-sniffio==1.2.0
-snowballstemmer==2.1.0
 spacy==2.3.5
-Sphinx==3.5.1
-sphinx-rtd-theme==0.5.1
-sphinxcontrib-applehelp==1.0.2
-sphinxcontrib-devhelp==1.0.2
-sphinxcontrib-htmlhelp==1.0.3
-sphinxcontrib-jsmath==1.0.1
-sphinxcontrib-qthelp==1.0.3
-sphinxcontrib-serializinghtml==1.1.4
-srsly==1.0.5
-statsmodels==0.12.2
 stopwordsiso==0.6.1
-tensorboard==2.4.1
-tensorboard-plugin-wit==1.8.0
 tensorflow==2.4.1
-tensorflow-estimator==2.4.0
-termcolor==1.1.0
-terminado==0.9.3
-testpath==0.4.4
-thinc==7.4.5
-threadpoolctl==2.1.0
-tokenizers==0.10.1
-toml==0.10.1
-torch==1.7.1
-tornado==6.1
 tqdm==4.56.1
-traitlets==5.0.5
-transformers==4.3.2
-twine==3.3.0
-typed-ast==1.4.2
-typing-extensions==3.7.4.3
-urllib3==1.24.3
-wasabi==0.8.2
-wcwidth==0.2.5
-webencodings==0.5.1
-Werkzeug==1.0.1
-wheel==0.36.2
-widgetsnbextension==3.5.1
 wordcloud==1.8.1
-wrapt==1.12.1
 xlrd==2.0.1
-yarl==1.5.1
-zipp==3.4.0
diff --git a/setup.py b/setup.py
index 436443d..a91d393 100644
--- a/setup.py
+++ b/setup.py
@@ -61,6 +61,7 @@
         "IPython",
         "sentence-transformers",
         "tqdm",
+        "defusedxml",
     ]
 
 if __name__ == "__main__":


Test patch:
diff --git a/tests/test_model.py b/tests/test_model.py
index 88771ab..ef4518d 100644
--- a/tests/test_model.py
+++ b/tests/test_model.py
@@ -34,27 +34,27 @@ def test_extract_frequent_kws(long_text_corpus):
     ]
 
 
-def test_translate_kw_output(long_text_corpus):
-    kws = model.extract_kws(
-        method="frequency",
-        text_corpus=long_text_corpus,
-        input_language="english",
-        output_language="german",
-        num_keywords=10,
-        prompt_remove_words=False,
-    )
-    assert kws == [
-        "VirginAmerica.",
-        "Flug",
-        "tco.",
-        "Carrie underwood",
-        "Lady Gaga",
-        "Fliege",
-        "virginamerica_ladygaga.",
-        "virginamerica_ladygaga_carrieunderwood.",
-        "Sitz",
-        "lax",
-    ]
+# def test_translate_kw_output(long_text_corpus):
+#     kws = model.extract_kws(
+#         method="frequency",
+#         text_corpus=long_text_corpus,
+#         input_language="english",
+#         output_language="german",
+#         num_keywords=10,
+#         prompt_remove_words=False,
+#     )
+#     assert kws == [
+#         "VirginAmerica.",
+#         "Flug",
+#         "tco.",
+#         "Carrie underwood",
+#         "Lady Gaga",
+#         "Fliege",
+#         "virginamerica_ladygaga.",
+#         "virginamerica_ladygaga_carrieunderwood.",
+#         "Sitz",
+#         "lax",
+#     ]
 
 
 def test_extract_TFIDF_kws(long_text_corpus):