Repository: mie-lab/trackintel

Issue:
tracking quality for weekdays and speed up
would be nice to include `granularity="weekday"` in the "temporal_tracking_quality" function.

At the moment, `granularity="hour"` is quite slow because `_split_overlaps(df, granularity="hour")` iterates over the df and split one hour at a time. This could be enhanced by running `_split_overlaps(df, granularity="day")` first.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/mie-lab/trackintel/pull/164?src=pr&el=h1) Report
> Merging [#164](https://codecov.io/gh/mie-lab/trackintel/pull/164?src=pr&el=desc) (a3405d6) into [master](https://codecov.io/gh/mie-lab/trackintel/commit/fe2459248e5294c3f762af67062545591ef9bfc2?el=desc) (fe24592) will **increase** coverage by `0.44%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/mie-lab/trackintel/pull/164/graphs/tree.svg?width=650&height=150&src=pr&token=ITDZEEE1MO)](https://codecov.io/gh/mie-lab/trackintel/pull/164?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #164      +/-   ##
==========================================
+ Coverage   75.92%   76.37%   +0.44%     
==========================================
  Files          36       36              
  Lines        1537     1566      +29     
  Branches      277      283       +6     
==========================================
+ Hits         1167     1196      +29     
  Misses        290      290              
  Partials       80       80              
```


| [Impacted Files](https://codecov.io/gh/mie-lab/trackintel/pull/164?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [trackintel/analysis/tracking\_quality.py](https://codecov.io/gh/mie-lab/trackintel/pull/164/diff?src=pr&el=tree#diff-dHJhY2tpbnRlbC9hbmFseXNpcy90cmFja2luZ19xdWFsaXR5LnB5) | `97.46% <100.00%> (+1.46%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/mie-lab/trackintel/pull/164?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/mie-lab/trackintel/pull/164?src=pr&el=footer). Last update [fe24592...a3405d6](https://codecov.io/gh/mie-lab/trackintel/pull/164?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/trackintel/analysis/tracking_quality.py b/trackintel/analysis/tracking_quality.py
index 6b7ec9b..e1848b2 100644
--- a/trackintel/analysis/tracking_quality.py
+++ b/trackintel/analysis/tracking_quality.py
@@ -1,33 +1,43 @@
 import datetime
 
 import pandas as pd
+import numpy as np
 
 
 def temporal_tracking_quality(source, granularity="all"):
     """
-    Calculate per-user temporal tracking quality.
+    Calculate per-user temporal tracking quality defined as the temporal coverage.
 
     Parameters
     ----------
     df : GeoDataFrame (as trackintel datamodels)
         The source dataframe to calculate temporal tracking quality.
 
-    granularity : {"all", "day", "hour"}, default="all"
-        The level of which the tracking quality is claculated. The default "all" returns
-        the overall tracking quality, and "day" and "hour" returns the tracking quality
-        by days and hours, respectively.
+    granularity : {"all", "day", "week", "weekday", "hour"}
+        The level of which the tracking quality is calculated. The default "all" returns
+        the overall tracking quality; "day" the tracking quality by days; "week" the quality
+        by weeks; "weekday" the quality by day of the week (e.g, Mondays, Tuesdays, etc.) and 
+        "hour" the quality by hours.
 
     Returns
     -------
     quality: DataFrame
-        A per-user temporal tracking quality dataframe.
+        A per-user per-granularity temporal tracking quality dataframe.
 
     Note
     ----
-    The temporal tracking quality is the time proportion of tracked period with the possible
-    time extent. The possible time extents of the different granularities are different: "all"
-    considers the time between the latest "finished_at" and the earliest "started_at", whereas
-    "day" considers the whole day (86400 sec) and "hour" considers the whole hour (3600 sec).
+    The temporal tracking quality is the ratio of tracking time and the total time extent. It is
+    calculated and returned per-user in the defined ``granularity``. The possible time extents of 
+    the different granularities are different:
+
+    - ``all`` considers the time between the latest "finished_at" and the earliest "started_at";
+    - ``week`` considers the whole week (604800 sec)
+    - ``day`` and ``weekday`` consider the whole day (86400 sec)
+    - ``hour`` considers the whole hour (3600 sec).
+
+    The tracking quality of each user is calculated based on his or her own tracking extent.
+    For granularity = ``day`` or ``week``, the quality["day"] or quality["week"] column displays the
+    time relative to the first record in the entire dataset.
 
     Examples
     --------
@@ -40,26 +50,100 @@ def temporal_tracking_quality(source, granularity="all"):
     df.reset_index(inplace=True)
     if granularity == "all":
         quality = df.groupby("user_id", as_index=False).apply(_get_tracking_quality_user, granularity)
+
     elif granularity == "day":
         # split records that span several days
         df = _split_overlaps(df, granularity=granularity)
         # get the tracked day relative to the first day
         start_date = df["started_at"].min().date()
         df["day"] = df["started_at"].apply(lambda x: (x.date() - start_date).days)
-        # calculate per-user per-day tracking quality
-        quality = df.groupby(["user_id", "day"], as_index=False).apply(_get_tracking_quality_user, granularity)
+
+        # calculate per-user per-day raw tracking quality
+        raw_quality = df.groupby(["user_id", "day"], as_index=False).apply(_get_tracking_quality_user, granularity)
+        # add quality = 0 records
+        quality = _get_all_quality(df, raw_quality, granularity)
+
+    elif granularity == "week":
+        # split records that span several days
+        df = _split_overlaps(df, granularity="day")
+        # get the tracked week relative to the first day
+        start_date = df["started_at"].min().date()
+        df["week"] = df["started_at"].apply(lambda x: (x.date() - start_date).days // 7)
+
+        # calculate per-user per-week raw tracking quality
+        raw_quality = df.groupby(["user_id", "week"], as_index=False).apply(_get_tracking_quality_user, granularity)
+        # add quality = 0 records
+        quality = _get_all_quality(df, raw_quality, granularity)
+
+    elif granularity == "weekday":
+        # split records that span several days
+        df = _split_overlaps(df, granularity="day")
+
+        # get the tracked week relative to the first day
+        start_date = df["started_at"].min().date()
+        df["week"] = df["started_at"].apply(lambda x: (x.date() - start_date).days // 7)
+        # get the weekday
+        df["weekday"] = df["started_at"].dt.weekday
+
+        # calculate per-user per-weekday raw tracking quality
+        raw_quality = df.groupby(["user_id", "weekday"], as_index=False).apply(_get_tracking_quality_user, granularity)
+        # add quality = 0 records
+        quality = _get_all_quality(df, raw_quality, granularity)
+
     elif granularity == "hour":
-        # split records that span several hours
-        df = _split_overlaps(df, granularity="hour")
+        # first do a day split to speed up the hour split
+        df = _split_overlaps(df, granularity="day")
+        df = _split_overlaps(df, granularity=granularity)
+
         # get the tracked day relative to the first day
         start_date = df["started_at"].min().date()
         df["day"] = df["started_at"].apply(lambda x: (x.date() - start_date).days)
         # get the hour
         df["hour"] = df["started_at"].dt.hour
-        # calculate per-user per-hour tracking quality
-        quality = df.groupby(["user_id", "hour"], as_index=False).apply(_get_tracking_quality_user, granularity)
+
+        # calculate per-user per-hour raw tracking quality
+        raw_quality = df.groupby(["user_id", "hour"], as_index=False).apply(_get_tracking_quality_user, granularity)
+        # add quality = 0 records
+        quality = _get_all_quality(df, raw_quality, granularity)
+
     else:
-        raise AttributeError(f"granularity unknown. We only support ['all', 'day', 'hour']. You passed {granularity}")
+        raise AttributeError(
+            f"granularity unknown. We only support ['all', 'day', 'week', 'weekday', 'hour']. You passed {granularity}"
+        )
+
+    return quality
+
+
+def _get_all_quality(df, raw_quality, granularity):
+    """
+    Add tracking quality values for empty bins.
+
+    raw_quality is calculated using `groupby` and does not report bins (=granularties) with
+    quality = 0. This function adds these values.
+
+    Parameters
+    ----------
+    df : GeoDataFrame (as trackintel datamodels)
+
+    raw_quality: DataFrame
+        The calculated raw tracking quality directly from the groupby operations.
+
+    granularity : {"all", "day", "weekday", "week", "hour"}
+        Used for accessing the column in raw_quality.
+
+    Returns
+    -------
+    quality: pandas.Series
+        A pandas.Series object containing the tracking quality
+    """
+    all_users = df["user_id"].unique()
+    all_granularity = np.arange(df[granularity].max() + 1)
+    # construct array containing all user and granularity combinations
+    all_combi = np.array(np.meshgrid(all_users, all_granularity)).T.reshape(-1, 2)
+    # the records with no corresponding raw_quality is nan, and transformed into 0
+    all_combi = pd.DataFrame(all_combi, columns=["user_id", granularity])
+    quality = all_combi.merge(raw_quality, how="left", on=["user_id", granularity], validate="one_to_one")
+    quality.fillna(0, inplace=True)
     return quality
 
 
@@ -72,9 +156,9 @@ def _get_tracking_quality_user(df, granularity="all"):
     df : GeoDataFrame (as trackintel datamodels)
         The source dataframe
 
-    granularity : {"all", "day", "hour"}, default "all"
+    granularity : {"all", "day", "weekday", "week", "hour"}, default "all"
         Determines the extent of the tracking. "all" the entire tracking period,
-        "day" a whole day and "hour" a whole hour.
+        "day" and "weekday" a whole day, "week" a whole week, and "hour" a whole hour.
 
     Returns
     -------
@@ -88,10 +172,21 @@ def _get_tracking_quality_user(df, granularity="all"):
     elif granularity == "day":
         # total seconds in a day
         extent = 60 * 60 * 24
+    elif granularity == "weekday":
+        # total seconds in an day * number of tracked weeks
+        # (entries from multiple weeks may be grouped together)
+        extent = 60 * 60 * 24 * (df["week"].max() - df["week"].min() + 1)
+    elif granularity == "week":
+        # total seconds in a week
+        extent = 60 * 60 * 24 * 7
     elif granularity == "hour":
-        # total seconds in an hour * number of different days
+        # total seconds in an hour * number of tracked days
         # (entries from multiple days may be grouped together)
-        extent = (60 * 60) * len(df["day"].unique())
+        extent = (60 * 60) * (df["day"].max() - df["day"].min() + 1)
+    else:
+        raise AttributeError(
+            f"granularity unknown. We only support ['all', 'day', 'week', 'weekday', 'hour']. You passed {granularity}"
+        )
     return pd.Series([tracked_duration / extent], index=["quality"])
 
 
@@ -105,13 +200,13 @@ def _split_overlaps(source, granularity="day"):
         The source to perform the split
 
     granularity : {'day', 'hour'}, default 'day'
-        The criteria of spliting. "day" splits records that have duration of several
+        The criteria of splitting. "day" splits records that have duration of several
         days and "hour" splits records that have duration of several hours.
 
     Returns
     -------
     GeoDataFrame (as trackintel datamodels)
-        The GeoDataFrame object after the spliting
+        The GeoDataFrame object after the splitting
     """
     df = source.copy()
     change_flag = __get_split_index(df, granularity=granularity)


Test patch:
diff --git a/tests/analysis/test_tracking_quality.py b/tests/analysis/test_tracking_quality.py
index b71f0e5..768d77a 100644
--- a/tests/analysis/test_tracking_quality.py
+++ b/tests/analysis/test_tracking_quality.py
@@ -54,6 +54,50 @@ class TestTemporal_tracking_quality:
 
         assert quality_manual == quality.loc[(quality["user_id"] == 0) & (quality["day"] == 0), "quality"].values[0]
 
+    def test_tracking_quality_week(self, testdata_stps_tpls_geolife_long):
+        """Test if the calculated tracking quality per week is correct."""
+        stps_tpls = testdata_stps_tpls_geolife_long
+
+        splitted = ti.analysis.tracking_quality._split_overlaps(stps_tpls, granularity="day")
+
+        # get the day relative to the start day
+        start_date = splitted["started_at"].min().date()
+        splitted["week"] = splitted["started_at"].apply(lambda x: (x.date() - start_date).days // 7)
+
+        # calculate tracking quality of the first week for the first user
+        user_0 = splitted.loc[splitted["user_id"] == 0]
+        extent = 60 * 60 * 24 * 7
+        tracked = (user_0["finished_at"] - user_0["started_at"]).dt.total_seconds().sum()
+        quality_manual = tracked / extent
+
+        # test if the result of the user agrees
+        quality = ti.analysis.tracking_quality.temporal_tracking_quality(stps_tpls, granularity="week")
+
+        assert quality_manual == quality.loc[(quality["user_id"] == 0), "quality"].values[0]
+
+    def test_tracking_quality_weekday(self, testdata_stps_tpls_geolife_long):
+        """Test if the calculated tracking quality per weekday is correct."""
+        stps_tpls = testdata_stps_tpls_geolife_long
+
+        splitted = ti.analysis.tracking_quality._split_overlaps(stps_tpls, granularity="day")
+
+        # get the day relative to the start day
+        start_date = splitted["started_at"].min().date()
+        splitted["week"] = splitted["started_at"].apply(lambda x: (x.date() - start_date).days // 7)
+
+        splitted["weekday"] = splitted["started_at"].dt.weekday
+
+        # calculate tracking quality of the first week for the first user
+        user_0 = splitted.loc[(splitted["user_id"] == 0) & (splitted["weekday"] == 3)]
+        extent = (60 * 60 * 24) * (user_0["week"].max() - user_0["week"].min() + 1)
+        tracked = (user_0["finished_at"] - user_0["started_at"]).dt.total_seconds().sum()
+        quality_manual = tracked / extent
+
+        # test if the result of the user agrees
+        quality = ti.analysis.tracking_quality.temporal_tracking_quality(stps_tpls, granularity="weekday")
+
+        assert quality_manual == quality.loc[(quality["user_id"] == 0) & (quality["weekday"] == 3), "quality"].values[0]
+
     def test_tracking_quality_hour(self, testdata_stps_tpls_geolife_long):
         """Test if the calculated tracking quality per hour is correct."""
         stps_tpls = testdata_stps_tpls_geolife_long
@@ -68,7 +112,7 @@ class TestTemporal_tracking_quality:
 
         # calculate tracking quality of an hour for the first user
         user_0 = splitted.loc[(splitted["user_id"] == 0) & (splitted["hour"] == 2)]
-        extent = (60 * 60) * len(user_0["day"].unique())
+        extent = (60 * 60) * (user_0["day"].max() - user_0["day"].min() + 1)
         tracked = (user_0["finished_at"] - user_0["started_at"]).dt.total_seconds().sum()
         quality_manual = tracked / extent
 
@@ -78,13 +122,23 @@ class TestTemporal_tracking_quality:
         assert quality_manual == quality.loc[(quality["user_id"] == 0) & (quality["hour"] == 2), "quality"].values[0]
 
     def test_tracking_quality_error(self, testdata_stps_tpls_geolife_long):
-        """Test if the an error is raised when passing unknown 'granularity'."""
+        """Test if the an error is raised when passing unknown 'granularity' to temporal_tracking_quality()."""
         stps_tpls = testdata_stps_tpls_geolife_long
 
         with pytest.raises(AttributeError):
             ti.analysis.tracking_quality.temporal_tracking_quality(stps_tpls, granularity=12345)
         with pytest.raises(AttributeError):
             ti.analysis.tracking_quality.temporal_tracking_quality(stps_tpls, granularity="random")
+            
+    def test_tracking_quality_user_error(self, testdata_stps_tpls_geolife_long):
+        """Test if the an error is raised when passing unknown 'granularity' to _get_tracking_quality_user()."""
+        stps_tpls = testdata_stps_tpls_geolife_long
+        user_0 = stps_tpls.loc[stps_tpls['user_id'] == 0] 
+
+        with pytest.raises(AttributeError):
+            ti.analysis.tracking_quality._get_tracking_quality_user(user_0, granularity=12345)
+        with pytest.raises(AttributeError):
+            ti.analysis.tracking_quality._get_tracking_quality_user(user_0, granularity="random")
 
     def test_split_overlaps_days(self, testdata_stps_tpls_geolife_long):
         """Test if _split_overlaps() function can split records that span several days."""
diff --git a/tests/preprocessing/test_positionfixes.py b/tests/preprocessing/test_positionfixes.py
index 77a40d8..cc79384 100644
--- a/tests/preprocessing/test_positionfixes.py
+++ b/tests/preprocessing/test_positionfixes.py
@@ -4,6 +4,8 @@ import sys
 from math import radians
 
 import geopandas as gpd
+from geopandas.testing import assert_geodataframe_equal
+
 import numpy as np
 import pandas as pd
 import pytest
@@ -148,8 +150,8 @@ class TestGenerate_triplegs():
 
         pfs, stps = geolife_pfs_stps_short
         pfs, tpls = pfs.as_positionfixes.generate_triplegs(stps, method='between_staypoints')
-
-        _assert_geodataframe_equal(tpls, testdata_tpls_geolife)
+        
+        assert_geodataframe_equal(tpls, testdata_tpls_geolife, check_less_precise=True)
 
     def test_generate_triplegs_case2(self, geolife_pfs_stps_short, testdata_tpls_geolife):
         """Test tripleg generation using the 'between_staypoints' method for case 2.
@@ -158,8 +160,8 @@ class TestGenerate_triplegs():
         pfs, stps = geolife_pfs_stps_short
         pfs.drop('staypoint_id', axis=1, inplace=True)
         pfs, tpls = pfs.as_positionfixes.generate_triplegs(stps, method='between_staypoints')
-
-        _assert_geodataframe_equal(tpls, testdata_tpls_geolife)
+        
+        assert_geodataframe_equal(tpls, testdata_tpls_geolife, check_less_precise=True)
 
     def test_generate_triplegs_case3(self, geolife_pfs_stps_short, testdata_tpls_geolife):
         """Test tripleg generation using the 'between_staypoints' method for case 3.
@@ -167,8 +169,8 @@ class TestGenerate_triplegs():
 
         pfs, _ = geolife_pfs_stps_short
         pfs, tpls = pfs.as_positionfixes.generate_triplegs(method='between_staypoints')
-
-        _assert_geodataframe_equal(tpls, testdata_tpls_geolife)
+        
+        assert_geodataframe_equal(tpls, testdata_tpls_geolife, check_less_precise=True)
 
     def test_tripleg_ids_in_positionfixes_case1(self, geolife_pfs_stps_short, testdata_pfs_ids_geolife):
         """
@@ -178,8 +180,8 @@ class TestGenerate_triplegs():
         """
         pfs, stps = geolife_pfs_stps_short
         pfs, _ = pfs.as_positionfixes.generate_triplegs(stps, method='between_staypoints')
-
-        _assert_geodataframe_equal(pfs, testdata_pfs_ids_geolife)
+        
+        assert_geodataframe_equal(pfs, testdata_pfs_ids_geolife, check_less_precise=True, check_like=True)
 
     def test_tripleg_ids_in_positionfixes_case2(self, geolife_pfs_stps_short, testdata_pfs_ids_geolife):
         """
@@ -192,7 +194,7 @@ class TestGenerate_triplegs():
         pfs, _ = pfs.as_positionfixes.generate_triplegs(stps, method='between_staypoints')
 
         testdata_pfs_ids_geolife.drop('staypoint_id', axis=1, inplace=True)
-        _assert_geodataframe_equal(pfs, testdata_pfs_ids_geolife)
+        assert_geodataframe_equal(pfs, testdata_pfs_ids_geolife, check_less_precise=True, check_like=True)
 
     def test_tripleg_ids_in_positionfixes_case3(self, geolife_pfs_stps_short, testdata_pfs_ids_geolife):
         """
@@ -201,8 +203,8 @@ class TestGenerate_triplegs():
         """
         pfs, _ = geolife_pfs_stps_short
         pfs, _ = pfs.as_positionfixes.generate_triplegs(method='between_staypoints')
-
-        _assert_geodataframe_equal(pfs, testdata_pfs_ids_geolife)
+        
+        assert_geodataframe_equal(pfs, testdata_pfs_ids_geolife, check_less_precise=True, check_like=True)
 
     def test_tripleg_generation_case2_empty_staypoints(self, geolife_pfs_stps_short, testdata_pfs_ids_geolife):
         """
@@ -228,11 +230,12 @@ class TestGenerate_triplegs():
         pfs_case2, tpls_case2 = pfs.drop('staypoint_id',
                                          axis=1).as_positionfixes.generate_triplegs(stps, method='between_staypoints')
         pfs_case3, tpls_case3 = pfs.as_positionfixes.generate_triplegs(method='between_staypoints')
-
-        _assert_geodataframe_equal(pfs_case1.drop('staypoint_id', axis=1), pfs_case2)
-        _assert_geodataframe_equal(pfs_case1, pfs_case3)
-        _assert_geodataframe_equal(tpls_case1, tpls_case2)
-        _assert_geodataframe_equal(tpls_case1, tpls_case3)
+        
+        assert_geodataframe_equal(pfs_case1.drop('staypoint_id', axis=1), pfs_case2, check_less_precise=True, check_like=True)
+        assert_geodataframe_equal(pfs_case1, pfs_case3, check_less_precise=True, check_like=True)
+        
+        assert_geodataframe_equal(tpls_case1, tpls_case2, check_less_precise=True, check_like=True)
+        assert_geodataframe_equal(tpls_case1, tpls_case3, check_less_precise=True, check_like=True)
 
     def test_generate_triplegs_dtype_consistent(self, pfs_geolife):
         """Test the dtypes for the generated columns."""
@@ -272,7 +275,6 @@ class TestGenerate_triplegs():
         The dataframe is sorted by date, then we check if the staypoint/tripleg from the row before was finished when
         the next one started.
         """
-        pfs_file = os.path.join('tests', 'data', 'geolife_long')
         pfs = pfs_geolife_long
         pfs, stps = pfs.as_positionfixes.generate_staypoints(method='sliding',
                                                              dist_threshold=25,
@@ -434,11 +436,3 @@ def _generate_staypoints_original(positionfixes, method='sliding',
     ret_stps['user_id'] = ret_stps['user_id'].astype(ret_pfs['user_id'].dtype)
 
     return ret_pfs, ret_stps
-
-
-def _assert_geodataframe_equal(gdf1, gdf2, **kwargs):
-    """Assert equal geometry and data seperatly for two geodataframes."""
-    assert (gdf1.geometry.geom_almost_equals(gdf2.geometry)).all()
-    pd.testing.assert_frame_equal(gdf1.drop(gdf1.geometry.name, axis=1),
-                                  gdf2.drop(gdf1.geometry.name, axis=1),
-                                  **kwargs)