Repository: Learning-and-Intelligent-Systems/lisdf (language: python)

Issue:
Investigate setup.py behavior
I was trying to `pip install lisdf@git+https://github.com/Learning-and-Intelligent-Systems/lisdf.git`

It was super slow, and it seemed like it was maybe trying to pull the lisdf-models sub-module. 

Here's the folder size as it was downloading and setting up. For comparison, the .git folder was only 7.4MB on a fresh clone from main and the lisdf-models directory should be empty if we are not pulling the submodule. 

```
  Running command git submodule update --init --recursive -q

  Preparing metadata (setup.py) ... done
...
❯ du -hd1
112K	./tests
 20K	./scripts
4.0K	./.github
 74M	./lisdf-models
305M	./.git
396K	./lisdf
380M	.
```

Could be remnants of an old .git where we had all the models here? 


Discussion / hints:
yeah I think that it is installing the submodule. relevant: https://github.com/pypa/pip/issues/6374
the only workarounds I can think of:
1. make lisdf-models an optional dependency (like lint, mypy, etc.), rather than a submodule
2. list lisdf on pypi and actually do `pip install lisdf`

I don't know of many downsides to (1) except that it would require a little work. Downside of (2) is that we would have to regularly release versions on pypi whenever there are important updates to the git repo.
I think (1) is probably a better option for now since the codebase isn't super release ready.

How would we have it as an optional dependency? Right now it's not a python package just a collection of files
right, so we would have to add a setup.py to lisdf-models, which is hopefully easy enough (I can take a look at that now)
We could also extract the models used in the tests into the test directory. That way, we can remove the dependency entirely and only people who want to use the models will clone it.

Fix patch:
diff --git a/.flake8 b/.flake8
index 119e260..0539436 100644
--- a/.flake8
+++ b/.flake8
@@ -6,7 +6,6 @@ exclude = (?x)(
     | env/
     | .venv/
     | venv/
-    | lisdf-models/
     | pybullet-planning/
     | kitchen-worlds/
   )
\ No newline at end of file
diff --git a/.gitignore b/.gitignore
index dafefd0..76e8d6a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -137,8 +137,3 @@ dmypy.json
 
 # DS store is annoying
 **/.DS_Store
-
-# Add lisdf-models as gitignore even though its submodule
-# We've already committed the reference.
-# Black uses this to discover what to ignore as well so it's a small hack
-lisdf-models
diff --git a/.gitmodules b/.gitmodules
index 7803153..e69de29 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,4 +0,0 @@
-[submodule "lisdf-models"]
-	path = lisdf-models
-	url = https://github.com/Learning-and-Intelligent-Systems/lisdf-models
-	branch = main
diff --git a/.isort.cfg b/.isort.cfg
index 76f9d73..021cce3 100644
--- a/.isort.cfg
+++ b/.isort.cfg
@@ -1,3 +1,3 @@
 [settings]
 profile = black
-skip =.env,env,.venv,venv,lisdf-models,pybullet-planning,kitchen-worlds
+skip =.env,env,.venv,venv,pybullet-planning,kitchen-worlds
diff --git a/README.md b/README.md
index d467216..62db831 100644
--- a/README.md
+++ b/README.md
@@ -10,22 +10,8 @@ popular specs to our spec.
 
 ## Installation
 
-### Installing with `pip`
-
-**TLDR:** `pip install lisdf@git+https://github.com/Learning-and-Intelligent-Systems/lisdf.git`
-
-If you wish to use the `lisdf-models` repository that includes our model files, you will need install from source as described below.
-
-### Installing from Source
-
-1. Clone the repository using `git clone`. 
-2. If you wish to use the `lisdf-models` repository which includes our model files then run 
-   `git submodule update --init`
-   - **WARNING!** The models in this repo are ~600MB as of 2022-03-25.
-3. Follow the instructions below to install dependencies. If you plan on contributing or running the tests, then 
-   additionally follow the "Instructions for Contributing" section.
-
 ### Dependencies
+
 This repository requires Python 3.8+. We recommend you create a 
 [conda env](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) or 
 [virtual env](https://docs.python.org/3.8/library/venv.html) with an appropriate Python version before installing
@@ -34,14 +20,21 @@ the other dependencies.
 If you are creating a virtual environment within the project directory, then you might want to call it one of
 `.env, env, .venv, venv` as the code checks have been configured to exclude those directories.
 
-If you plan on contributing or running tests, follow the instructions in the next section. Otherwise:
-1. Run `pip install -e .` to install.
+### Installing with `pip`
+
+Run `pip install lisdf@git+https://github.com/Learning-and-Intelligent-Systems/lisdf.git` to install.
+
+### Installing from Source
+
+1. Clone the repository using `git clone`. 
+2. Run `pip install -e .` to install.
 
 ## Instructions For Contributing
 
 ### Dev Dependencies
-Run `pip install -e .[develop]` to install all dependencies for development/contribution.
-You should also pull the `lisdf-models` submodule if you haven't already (see Installation section above).
+Clone the repository using `git clone`. Then run `pip install -e '.[develop]'` to install all dependencies for development/contribution.
+
+Warning: the dev dependencies include the [`lisdf-models` repository](https://github.com/Learning-and-Intelligent-Systems/lisdf-models) which is ~700MB big as of 10th September 2022.
 
 ### Pushing your Changes
 You can't directly push to the `main` branch.  All contributions will require review to ensure code is understandable 
diff --git a/lisdf-models b/lisdf-models
deleted file mode 160000
index 82fcf0b..0000000
--- a/lisdf-models
+++ /dev/null
@@ -1,1 +0,0 @@
-Subproject commit 82fcf0b2c16368ccc436b271abbda8925ae52d7f
diff --git a/lisdf/parsing/parse_sdf.py b/lisdf/parsing/parse_sdf.py
index 47f79f1..8110f3a 100644
--- a/lisdf/parsing/parse_sdf.py
+++ b/lisdf/parsing/parse_sdf.py
@@ -60,6 +60,9 @@ def load_sdf(model_name: str, models_dir: str) -> SDF:
 
 
 if __name__ == "__main__":  # pragma: no cover
+    import lisdf_models
+
+    models_dir = os.path.dirname(lisdf_models.__file__)
     sdf_test = "mud_test/model.sdf"
-    sdf_results = load_sdf(sdf_test, models_dir="lisdf-models/models/")
+    sdf_results = load_sdf(sdf_test, models_dir=models_dir)
     # print(sdf_results)
diff --git a/mypy.ini b/mypy.ini
index bddb486..104bd16 100644
--- a/mypy.ini
+++ b/mypy.ini
@@ -7,7 +7,6 @@ exclude = (?x)(
     | env/
     | .venv/
     | venv/
-    | lisdf-models/
     | pybullet-planning/
     | kitchen-worlds/
   )
diff --git a/setup.py b/setup.py
index ec26780..97da048 100644
--- a/setup.py
+++ b/setup.py
@@ -10,8 +10,7 @@
         "numpy",
         "lark",
     ],
-    include_package_data=True,
-    package_data={"lisdf": ["py.typed"]},  # for mypy
+    package_data={"lisdf": ["lisdf/**"]},  # mypy and pddl files
     extras_require={
         "develop": [
             # Formatting
@@ -27,6 +26,8 @@
             "pytest",
             "pytest-cov",
             "mock",
+            # LISdf models
+            "lisdf_models@git+https://github.com/Learning-and-Intelligent-Systems/lisdf-models.git",  # noqa: E501
         ]
     },
 )


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index 8596050..e272c6d 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,21 +1,14 @@
 import os
 
+import lisdf_models
 import pytest
 
-_LISDF_MODELS = os.path.join("lisdf-models", "models")
-
 
 @pytest.fixture(scope="session")
 def models_dir() -> str:
-    """Determine models directory for the lisdf-models submodule"""
-    # e.g. /home/user/lisdf/tests
-    test_dir = os.path.dirname(__file__)
-
-    # e.g. /home/user/lisdf
-    lisdf_dir = os.path.dirname(test_dir)
-
+    """Determine models directory for the lisdf-models"""
     # e.g./home/user/lisdf/lisdf-models/models
-    lisdf_models_dir = os.path.join(lisdf_dir, _LISDF_MODELS)
+    lisdf_models_dir = os.path.dirname(lisdf_models.__file__)
 
     # Check if lisdf-models actually exists
     if not os.path.exists(lisdf_models_dir):
diff --git a/tests/test_parsing/test_parse_sdf.py b/tests/test_parsing/test_parse_sdf.py
index 3d62d7a..52eeb74 100644
--- a/tests/test_parsing/test_parse_sdf.py
+++ b/tests/test_parsing/test_parse_sdf.py
@@ -21,7 +21,7 @@ def test_recursive_sdf_parsing(models_dir):
 
 
 def test_m0m(models_dir):
-    parsed_sdf = load_sdf("m0m/model.sdf", models_dir)
+    parsed_sdf = load_sdf("m0m/model_no_pr2.sdf", models_dir)
     world = parsed_sdf.aggregate_order[0]
     assert world.models[4].links[0].collisions[0].geometry.uri == os.path.join(
         models_dir, "m0m/../ycb/011_banana/textured.obj"