Repository: niconoe/pyinaturalist (python)

Issue:
Add user life list endpoint
iNaturalist life lists ([example](https://www.inaturalist.org/lifelists/jkcook?view=tree&details_view=observations)) use an undocumented endpoint, `GET /observations/taxonomy`, which looks quite useful. This provides a much faster and more concise way to create a taxonomy tree of your own observations, which is something I've previously done the hard way with the main `/observations` endpoint.

Example results:
```json
{
    "count_without_taxon": 4,
    "results": [
        {
            "count": 1,
            "descendant_obs_count": 3023,
            "direct_obs_count": 1,
            "id": 1,
            "is_active": true,
            "name": "Animalia",
            "parent_id": 48460,
            "rank": "kingdom",
            "rank_level": 70
        },
        {
            "descendant_obs_count": 447,
            "direct_obs_count": 0,
            "id": 2,
            "is_active": true,
            "name": "Chordata",
            "parent_id": 1,
            "rank": "phylum",
            "rank_level": 60
        },
        {
            "count": 4,
            "descendant_obs_count": 340,
            "direct_obs_count": 4,
            "id": 3,
            "is_active": true,
            "name": "Aves",
            "parent_id": 355675,
            "rank": "class",
            "rank_level": 50
        }
  ]
}
```

Related thread: https://forum.inaturalist.org/t/create-api-endpoint-to-count-leaf-node-taxa-per-observer-not-just-species

Pull request description:
Add user life list endpoint (GET /observations/taxonomy) + models
Closes #159 

Patch:
diff --git a/HISTORY.md b/HISTORY.md
index d6e9abe..e2d476b 100644
--- a/HISTORY.md
+++ b/HISTORY.md
@@ -5,6 +5,7 @@
 
 ### New Endpoints
 * Added new function for **Observation sounds** endpoint: `upload_sounds()`
+* Added new function for **Life list** endpoint: `get_observation_taxonomy()`
 
 ### Modified Endpoints
 * Added a `photos` parameter `create_observation()` and `update_observation()` to upload photos
@@ -20,9 +21,26 @@
     * Added aliases for backwards-compatibility, so imports from `pyinaturalist.rest_api` and
       `pyinaturalist.node_api` will still work, but these are deprecated and will be removed in a future release
 
+### Models
+Added data models for all API response types, to support working with typed python objects instead of JSON:
+* Annotation
+* Comment
+* Identification
+* LifeList
+    * LifeListTaxon
+* Observation
+* ObservationField
+    * ObservationFieldValue
+* Photo
+* Place
+* Project
+    * ProjectObservationField
+    * ProjectUser
+* Taxon
+* User
+
 ### Other Changes
 * Set up pyinaturalist-notebook to [runnable with Binder](https://mybinder.org/v2/gh/niconoe/pyinaturalist/main?filepath=examples)
-* Added data models for all API response types, to support working with typed python objects instead of JSON
 
 -----
 ## 0.13 (2021-05-22)
diff --git a/docs/endpoints.rst b/docs/endpoints.rst
index aa9b4b2..8980fd2 100644
--- a/docs/endpoints.rst
+++ b/docs/endpoints.rst
@@ -29,10 +29,10 @@ All iNaturalist endpoints
 
 .. note::
 
-    The two iNaturalist APIs expose a combined total of 102 endpoints. Some of these are generally
+    The two iNaturalist APIs expose a combined total of 104+ endpoints. Some of these are generally
     useful and could potentially be added to pyinaturalist, but many others are primarily for
-    internal use by the iNaturalist web application and mobile apps, and are unlikely to be added
-    unless there are specific use cases for them.
+    internal use by iNaturalist.org and its mobile apps, and are unlikely to be added unless
+    specifically requested.
 
 .. mdinclude:: endpoints_table.md
     :start-line: 1
diff --git a/docs/endpoints_table.md b/docs/endpoints_table.md
index 17e1ef8..5b87ba1 100644
--- a/docs/endpoints_table.md
+++ b/docs/endpoints_table.md
@@ -14,15 +14,15 @@ DELETE            | /votes/unvote/annotation/{id}               |
 POST              | /comments                                   |
 DELETE            | /comments/{id}                              |
 PUT               | /comments/{id}                              |
-GET               | /controlled_terms                           | yes
-GET               | /controlled_terms/for_taxon                 | yes
+GET               | /controlled_terms                           | ✔️
+GET               | /controlled_terms/for_taxon                 | ✔️
 POST              | /flags                                      |
 DELETE            | /flags/{id}                                 |
 PUT               | /flags/{id}                                 |
 DELETE            | /identifications/{id}                       |
-GET               | /identifications/{id}                       | yes
+GET               | /identifications/{id}                       | ✔️
 PUT               | /identifications/{id}                       |
-GET               | /identifications                            | yes
+GET               | /identifications                            | ✔️
 POST              | /identifications                            |
 GET               | /identifications/categories                 |
 GET               | /identifications/species_counts             |
@@ -42,7 +42,7 @@ DELETE            | /observation_photos/{id}                    |
 PUT               | /observation_photos/{id}                    |
 POST              | /observation_photos                         |
 DELETE            | /observations/{id}                          |
-GET               | /observations/{id}                          | yes
+GET               | /observations/{id}                          | ✔️
 PUT               | /observations/{id}                          |
 POST              | /observations/{id}/fave                     |
 DELETE            | /observations/{id}/unfave                   |
@@ -55,19 +55,20 @@ GET               | /observations/{id}/taxon_summary            |
 POST              | /subscriptions/observation/{id}/subscribe   |
 POST              | /votes/vote/observation/{id}                |
 DELETE            | /votes/unvote/observation/{id}              |
-GET               | /observations                               | yes
+GET               | /observations                               | ✔️
 POST              | /observations                               |
 GET               | /observations/deleted                       |
-GET               | /observations/histogram                     | yes
-GET               | /observations/identifiers                   | yes
-GET               | /observations/observers                     | yes
+GET               | /observations/histogram                     | ✔️
+GET               | /observations/identifiers                   | ✔️
+GET               | /observations/observers                     | ✔️
 GET               | /observations/popular_field_values          |
-GET               | /observations/species_counts                | yes
+GET               | /observations/species_counts                | ✔️
+GET               | /observations/taxonomy                      | ✔️
 GET               | /observations/updates                       |
 PUT               | /observations/{id}/viewed_updates           |
-GET               | /places/{id}                                | yes
-GET               | /places/autocomplete                        | yes
-GET               | /places/nearby                              | yes
+GET               | /places/{id}                                | ✔️
+GET               | /places/autocomplete                        | ✔️
+GET               | /places/nearby                              | ✔️
 GET               | /posts                                      |
 POST              | /posts                                      |
 DELETE            | /posts/{id}                                 |
@@ -76,8 +77,8 @@ GET               | /posts/for_user                             |
 DELETE            | /project_observations/{id}                  |
 PUT               | /project_observations/{id}                  |
 POST              | /project_observations                       |
-GET               | /projects                                   | yes
-GET               | /projects/{id}                              | yes
+GET               | /projects                                   | ✔️
+GET               | /projects/{id}                              | ✔️
 POST              | /projects/{id}/join                         |
 DELETE            | /projects/{id}/leave                        |
 GET               | /projects/{id}/members                      |
@@ -86,14 +87,14 @@ POST              | /projects/{id}/add                          |
 DELETE            | /projects/{id}/remove                       |
 GET               | /projects/autocomplete                      |
 POST              | /subscriptions/project/{id}/subscribe       |
-GET               | /search                                     | yes
-GET               | /taxa/{id}                                  | yes
-GET               | /taxa                                       | yes
-GET               | /taxa/autocomplete                          | yes
-GET               | /users/{id}                                 | yes
+GET               | /search                                     | ✔️
+GET               | /taxa/{id}                                  | ✔️
+GET               | /taxa                                       | ✔️
+GET               | /taxa/autocomplete                          | ✔️
+GET               | /users/{id}                                 | ✔️
 PUT               | /users/{id}                                 |
 GET               | /users/{id}/projects                        |
-GET               | /users/autocomplete                         | yes
+GET               | /users/autocomplete                         | ✔️
 GET               | /users/me                                   |
 DELETE            | /users/{id}/mute                            |
 POST              | /users/{id}/mute                            |
@@ -123,11 +124,11 @@ DELETE            | /comments/{id}                              |
 POST              | /identifications                            |
 PUT               | /identifications/{id}                       |
 DELETE            | /identifications/{id}                       |
-GET               | /observations                               | yes
-POST              | /observations                               | yes
-GET               | /observations/{id}                          | yes
-PUT               | /observations/{id}                          | yes
-DELETE            | /observations/{id}                          | yes
+GET               | /observations                               | ✔️
+POST              | /observations                               | ✔️
+GET               | /observations/{id}                          | ✔️
+PUT               | /observations/{id}                          | ✔️
+DELETE            | /observations/{id}                          | ✔️
 POST              | /observations/{id}/quality/:metric          |
 DELETE            | /observations/{id}/quality/:metric          |
 PUT               | /observations/{id}/viewed_updates           |
@@ -135,12 +136,12 @@ GET               | /observations/:username                     |
 GET               | /observations/project/{id}                  |
 GET               | /observations/taxon_stats                   |
 GET               | /observations/user_stats                    |
-GET               | /observation_fields                         | yes
+GET               | /observation_fields                         | ✔️
 POST              | /observation_field_values                   |
-PUT               | /observation_field_values/{id}              | yes
+PUT               | /observation_field_values/{id}              | ✔️
 DELETE            | /observation_field_values/{id}              |
-POST              | /observation_photos                         | yes
-POST              | /observation_sounds                         | yes
+POST              | /observation_photos                         | ✔️
+POST              | /observation_sounds                         | ✔️
 GET               | /places                                     |
 GET               | /projects                                   |
 GET               | /projects/{id}                              |
diff --git a/examples/sample_responses.py b/examples/sample_responses.py
index b140867..86f5947 100755
--- a/examples/sample_responses.py
+++ b/examples/sample_responses.py
@@ -37,6 +37,7 @@ def load_sample_data(filename):
 obs_json = load_sample_data('get_observations_node_page1.json')['results'][0]
 obs_json_ofvs = load_sample_data('get_observation_with_ofvs.json')['results'][0]
 obs_field_json = load_sample_data('get_observation_fields_page1.json')[0]
+obs_taxonomy_json = load_sample_data('get_observation_taxonomy.json')
 place_json = load_sample_data('get_places_by_id.json')['results'][0]
 places_nearby_json = load_sample_data('get_places_nearby.json')['results']
 project_json = load_sample_data('get_projects_obs_fields.json')['results'][0]
@@ -57,6 +58,7 @@ photo_json = taxon_json['default_photo']
 annotation = Annotation.from_json(annotation_json)
 comment = Comment.from_json(comment_json)
 identification = Identification.from_json(identification_json)
+life_list = LifeList.from_taxonomy_json(obs_taxonomy_json)
 observation = Observation.from_json(obs_json)
 observation_with_ofvs = Observation.from_json(obs_json_ofvs)
 observation_field = ObservationField.from_json(obs_field_json)
diff --git a/pyinaturalist/models/__init__.py b/pyinaturalist/models/__init__.py
index 4e617d8..be54d2c 100644
--- a/pyinaturalist/models/__init__.py
+++ b/pyinaturalist/models/__init__.py
@@ -12,6 +12,7 @@ from pyinaturalist.models.base import (
     coordinate_pair,
     get_model_fields,
     kwarg,
+    load_json,
     make_attribute,
 )
 from pyinaturalist.models.lazy_property import LazyProperty, add_lazy_attrs
@@ -26,6 +27,7 @@ from pyinaturalist.models.user import User
 from pyinaturalist.models.annotation import Annotation
 from pyinaturalist.models.comment import Comment
 from pyinaturalist.models.identification import Identification
+from pyinaturalist.models.life_list import LifeList, LifeListTaxon
 from pyinaturalist.models.place import Place
 from pyinaturalist.models.project import Project, ProjectObservationField, ProjectUser
 from pyinaturalist.models.observation_field import ObservationField, ObservationFieldValue
diff --git a/pyinaturalist/models/base.py b/pyinaturalist/models/base.py
index ef113ee..e31df03 100644
--- a/pyinaturalist/models/base.py
+++ b/pyinaturalist/models/base.py
@@ -8,7 +8,7 @@ from typing import Any, Dict, Iterable, List, Optional, Type, TypeVar, Union
 
 from attr import Attribute, asdict, define, field, fields_dict
 
-from pyinaturalist.constants import AnyFile, ResponseOrFile
+from pyinaturalist.constants import AnyFile, JsonResponse, ResponseOrFile
 from pyinaturalist.response_format import convert_lat_long, try_datetime
 
 FIELD_DEFAULTS = {
@@ -99,8 +99,10 @@ def get_model_fields(obj: Any) -> Iterable[Attribute]:
     return attrs
 
 
-def load_json(value: AnyFile):
+def load_json(value: Union[JsonResponse, AnyFile]):
     """Load JSON from a file path or file-like object"""
+    if isinstance(value, dict):
+        return value
     if isinstance(value, (Path, str)):
         with open(expanduser(str(value))) as f:
             return json.load(f)
diff --git a/pyinaturalist/models/life_list.py b/pyinaturalist/models/life_list.py
new file mode 100644
index 0000000..7e55ccd
--- /dev/null
+++ b/pyinaturalist/models/life_list.py
@@ -0,0 +1,57 @@
+from typing import Dict, List, Optional
+
+from attr import field
+
+from pyinaturalist.constants import ResponseOrFile
+from pyinaturalist.models import BaseModel, define_model, kwarg, load_json
+
+
+@define_model
+class LifeListTaxon(BaseModel):
+    """A dataclass containing information about a single taxon in a user's life list"""
+
+    count: int = kwarg
+    descendant_obs_count: int = kwarg
+    direct_obs_count: int = kwarg
+    id: int = kwarg
+    is_active: bool = kwarg
+    name: str = kwarg
+    parent_id: int = kwarg
+    rank_level: int = kwarg
+    rank: str = kwarg  # Enum
+
+
+@define_model
+class LifeList(BaseModel):
+    """A dataclass containing information about a user's life list, based on the schema of
+    ``GET /observations/taxonomy``
+    """
+
+    count_without_taxon: int = field(default=0)
+    taxa: List[LifeListTaxon] = field(factory=list, converter=LifeListTaxon.from_json_list)
+    user_id: int = kwarg
+    _taxon_counts: Dict[int, int] = field(default=None, init=False, repr=False)
+
+    @classmethod
+    def from_taxonomy_json(cls, value: ResponseOrFile, user_id: int = None) -> Optional['LifeList']:
+        json_value = load_json(value)
+        count_without_taxon = json_value.get('count_without_taxon', 0)
+        if 'results' in json_value:
+            json_value = json_value['results']
+
+        life_list_json = {'taxa': value, 'user_id': user_id, 'count_without_taxon': count_without_taxon}
+        return cls.from_json(life_list_json)  # type: ignore
+
+    def count(self, taxon_id: int) -> int:
+        """Get an observation count for the specified taxon and its descendants.
+        **Note:** ``-1`` can be used an alias for ``count_without_taxon``.
+        """
+        # Make and cache an index of taxon IDs and observation counts
+        if self._taxon_counts is None:
+            self._taxon_counts = {t.id: t.descendant_obs_count for t in self.taxa}
+            self._taxon_counts[-1] = self.count_without_taxon
+        return self._taxon_counts.get(taxon_id, 0)
+
+    # TODO: Restructure flat taxonomy list into a tree, based on parent_id
+    # def tree(self):
+    #     pass
diff --git a/pyinaturalist/v1/__init__.py b/pyinaturalist/v1/__init__.py
index cea9a10..efa1121 100644
--- a/pyinaturalist/v1/__init__.py
+++ b/pyinaturalist/v1/__init__.py
@@ -38,6 +38,7 @@ from pyinaturalist.v1.observations import (
     get_observation_identifiers,
     get_observation_observers,
     get_observation_species_counts,
+    get_observation_taxonomy,
     get_observations,
 )
 from pyinaturalist.v1.places import get_places_autocomplete, get_places_by_id, get_places_nearby
diff --git a/pyinaturalist/v1/observations.py b/pyinaturalist/v1/observations.py
index 3458ec6..d807a7a 100644
--- a/pyinaturalist/v1/observations.py
+++ b/pyinaturalist/v1/observations.py
@@ -1,4 +1,4 @@
-from typing import List
+from typing import List, Union
 
 from pyinaturalist import api_docs as docs
 from pyinaturalist.constants import NODE_OBS_ORDER_BY_PROPERTIES, HistogramResponse, JsonResponse
@@ -297,3 +297,29 @@ def get_observation_identifiers(**params) -> JsonResponse:
     )
     r.raise_for_status()
     return r.json()
+
+
+@add_paginate_all(method='page')
+def get_observation_taxonomy(user_id: Union[int, str], user_agent: str = None) -> JsonResponse:
+    """Get observation counts for all taxa in a full taxonomic tree. In the web UI, these are used
+    for life lists.
+
+    Args:
+        user_id: iNaturalist user ID or username
+
+    Example:
+        >>> response = get_observation_taxonomy(user_id='my_username')
+        ...
+
+        .. admonition:: Example Response
+            :class: toggle
+
+            .. literalinclude:: ../sample_data/get_observation_taxonomy.json
+                :language: JSON
+
+    Returns:
+        Response dict containing taxon records with counts
+    """
+    r = get_v1('observations/taxonomy', params={'user_id': user_id}, user_agent=user_agent)
+    r.raise_for_status()
+    return r.json()


Test patch:
diff --git a/test/sample_data/get_observation_taxonomy.json b/test/sample_data/get_observation_taxonomy.json
new file mode 100644
index 0000000..051dc7d
--- /dev/null
+++ b/test/sample_data/get_observation_taxonomy.json
@@ -0,0 +1,100 @@
+{
+  "count_without_taxon": 4,
+  "size": 9,
+  "results": [
+    {
+      "id": 1,
+      "count": 1,
+      "name": "Animalia",
+      "rank": "kingdom",
+      "rank_level": 70,
+      "is_active": true,
+      "parent_id": 48460,
+      "descendant_obs_count": 3023,
+      "direct_obs_count": 1
+    },
+    {
+      "id": 2,
+      "name": "Chordata",
+      "rank": "phylum",
+      "rank_level": 60,
+      "is_active": true,
+      "parent_id": 1,
+      "descendant_obs_count": 447,
+      "direct_obs_count": 0
+    },
+    {
+      "id": 3,
+      "count": 4,
+      "name": "Aves",
+      "rank": "class",
+      "rank_level": 50,
+      "is_active": true,
+      "parent_id": 355675,
+      "descendant_obs_count": 340,
+      "direct_obs_count": 4
+    },
+    {
+      "id": 573,
+      "name": "Galliformes",
+      "rank": "order",
+      "rank_level": 40,
+      "is_active": true,
+      "parent_id": 3,
+      "descendant_obs_count": 4,
+      "direct_obs_count": 0
+    },
+    {
+      "id": 574,
+      "name": "Phasianidae",
+      "rank": "family",
+      "rank_level": 30,
+      "is_active": true,
+      "parent_id": 573,
+      "descendant_obs_count": 4,
+      "direct_obs_count": 0
+    },
+    {
+      "id": 889,
+      "name": "Bonasa",
+      "rank": "genus",
+      "rank_level": 20,
+      "is_active": true,
+      "parent_id": 574,
+      "descendant_obs_count": 2,
+      "direct_obs_count": 0
+    },
+    {
+      "id": 890,
+      "count": 2,
+      "name": "Bonasa umbellus",
+      "rank": "species",
+      "rank_level": 10,
+      "is_active": true,
+      "parent_id": 889,
+      "descendant_obs_count": 2,
+      "direct_obs_count": 2
+    },
+    {
+      "id": 980,
+      "name": "Phasianus",
+      "rank": "genus",
+      "rank_level": 20,
+      "is_active": true,
+      "parent_id": 574,
+      "descendant_obs_count": 2,
+      "direct_obs_count": 0
+    },
+    {
+      "id": 981,
+      "count": 2,
+      "name": "Phasianus colchicus",
+      "rank": "species",
+      "rank_level": 10,
+      "is_active": true,
+      "parent_id": 980,
+      "descendant_obs_count": 2,
+      "direct_obs_count": 2
+    }
+ ]
+}
diff --git a/test/test_models.py b/test/test_models.py
index 52d840f..623476a 100644
--- a/test/test_models.py
+++ b/test/test_models.py
@@ -8,6 +8,8 @@ from pyinaturalist.models import (
     OFV,
     Annotation,
     Comment,
+    LifeList,
+    LifeListTaxon,
     Observation,
     ObservationField,
     Photo,
@@ -24,6 +26,7 @@ from test.conftest import load_sample_data, sample_data_path
 obs_json = load_sample_data('get_observations_node_page1.json')['results'][0]
 obs_json_ofvs = load_sample_data('get_observation_with_ofvs.json')['results'][0]
 obs_field_json = load_sample_data('get_observation_fields_page1.json')[0]
+obs_taxonomy_json = load_sample_data('get_observation_taxonomy.json')
 place_json = load_sample_data('get_places_by_id.json')['results'][0]
 places_nearby_json = load_sample_data('get_places_nearby.json')['results']
 project_json = load_sample_data('get_projects.json')['results'][0]
@@ -90,13 +93,11 @@ def test_annotation_values():
 
 
 def test_comment_converters():
-    """Test data type conversions"""
     comment = Comment.from_json(comment_json)
     assert isinstance(comment.user, User) and comment.user.id == 2852555
 
 
 def test_comment_empty():
-    """We should be able to initialize the model with no args, and get sane defaults"""
     comment = Comment()
     assert isinstance(comment.created_at, datetime)
     assert comment.user is None
@@ -118,6 +119,29 @@ def test_identification_empty():
     assert identification.user is None
 
 
+# Life Lists
+# --------------------
+
+
+def test_life_list_converters():
+    life_list = LifeList.from_taxonomy_json(obs_taxonomy_json)
+    assert isinstance(life_list.taxa[0], LifeListTaxon) and life_list.taxa[0].id == 1
+
+
+def test_life_list_empty():
+    life_list = LifeList()
+    life_list.taxa == []
+    life_list._taxon_counts is None
+
+
+def test_life_list_count():
+    life_list = LifeList.from_taxonomy_json(obs_taxonomy_json)
+    assert life_list.count(1) == 3023  # Animalia
+    assert life_list.count(981) == 2  # Phasianus colchicus
+    assert life_list.count(-1) == 4  # Observations with no taxon
+    assert life_list.count(9999999) == 0  # Unobserved taxon
+
+
 # Observations
 # --------------------
 
diff --git a/test/v1/test_observations.py b/test/v1/test_observations.py
index 2b6b210..79c8cd8 100644
--- a/test/v1/test_observations.py
+++ b/test/v1/test_observations.py
@@ -12,6 +12,7 @@ from pyinaturalist.v1 import (
     get_observation_identifiers,
     get_observation_observers,
     get_observation_species_counts,
+    get_observation_taxonomy,
     get_observations,
 )
 from test.conftest import load_sample_data
@@ -208,3 +209,18 @@ def test_get_observation_species_counts__all_pages(sleep, requests_mock):
 def test_get_observation_species_counts__invalid_multiple_choice_params():
     with pytest.raises(ValueError):
         get_observation_species_counts(quality_grade='None', iconic_taxa='slime molds')
+
+
+def test_get_observation_taxonomy(requests_mock):
+    requests_mock.get(
+        f'{API_V1_BASE_URL}/observations/taxonomy',
+        json=load_sample_data('get_observation_taxonomy.json'),
+        status_code=200,
+    )
+    response = get_observation_taxonomy(user_id=12345)
+    first_result = response['results'][0]
+
+    assert response['count_without_taxon'] == 4
+    assert first_result['id'] == 1
+    assert first_result['name'] == 'Animalia'
+    assert first_result['descendant_obs_count'] == 3023