anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: cmu-delphi/covidcast-indicators
Issue: delphi-utils dual-year geocoding files are not being exported in the package

While testing the jhu indicator on staging I discovered that it doesn't actually run:

```
[indicators@app-mono-dev-01 jhu]$ env/bin/python -m delphi_jhu
Traceback (most recent call last):
  File "/home/indicators/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/indicators/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/data/indicators/runtime/jhu/delphi_jhu/__main__.py", line 12, in <module>
    run_module(read_params())  # pragma: no cover
  File "/mnt/data/indicators/runtime/jhu/delphi_jhu/run.py", line 91, in run_module
    gmpr = GeoMapper()
  File "/mnt/data/indicators/runtime/jhu/env/lib/python3.8/site-packages/delphi_utils/geomap.py", line 136, in __init__
    self._load_crosswalk_from_file(from_code,
  File "/mnt/data/indicators/runtime/jhu/env/lib/python3.8/site-packages/delphi_utils/geomap.py", line 145, in _load_crosswalk_from_file
    stream = pkg_resources.resource_stream(__name__, data_path)
  File "/mnt/data/indicators/runtime/jhu/env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1150, in resource_stream
    return get_provider(package_or_requirement).get_resource_stream(
  File "/mnt/data/indicators/runtime/jhu/env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1613, in get_resource_stream
    return open(self._fn(self.module_path, resource_name), 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/data/indicators/runtime/jhu/env/lib/python3.8/site-packages/delphi_utils/data/2020/zip_fips_table.csv'
```

However I couldn't replicate it in my dev environment until I swapped the editable-install command used by `make test` for the uneditable-install command used by [build-and-package.sh](https://github.com/cmu-delphi/covidcast-indicators/blob/main/jenkins/build-and-package.sh).

Looks like we didn't update the package spec when we added in the 2010/2020 year split:

https://github.com/cmu-delphi/covidcast-indicators/blob/0bbcc1f91357318f2598d77c0a22bbe8fe0e02e8/_delphi_utils_python/setup.py#L43

CI didn't catch it because it always uses `pip install -e` 😭 , and it hasn't shown up as an error in production because Jenkins hasn't actually been deploying releases for quite some time. Thankfully Jenkins is fixed now, but we should:

- [ ] Fix the package spec (I'll take this)
- [x] #1443


Changed files:
_delphi_utils_python/setup.py

diff --git a/_delphi_utils_python/setup.py b/_delphi_utils_python/setup.py
index a486ead6..100e01e3 100644
--- a/_delphi_utils_python/setup.py
+++ b/_delphi_utils_python/setup.py
@@ -40,5 +40,5 @@ setup(
         "Programming Language :: Python :: 3.8",
     ],
     packages=find_packages(),
-    package_data={'': ['data/*.csv']}
+    package_data={'': ['data/20*/*.csv']}
 )
Repository: cmu-delphi/covidcast-indicators
Issue: USAFacts is failing checks in main

USAFacts is failing tests in main, probably due to a hamhanded conflict resolution I installed on Friday 😞 

Linter errors:
```
env/bin/pylint delphi_usafacts
************* Module delphi_usafacts.geo
delphi_usafacts/geo.py:28:42: E0602: Undefined variable 'STATE_TO_FIPS' (undefined-variable)
************* Module delphi_usafacts.pull
delphi_usafacts/pull.py:78:18: E0602: Undefined variable 'pop_df' (undefined-variable)
```

(these cause commensurate errors in tests as well)

Changed files:
usafacts/delphi_usafacts/pull.py, usafacts/delphi_usafacts/geo.py

diff --git a/usafacts/delphi_usafacts/geo.py b/usafacts/delphi_usafacts/geo.py
index c86bb8ee..87457162 100644
--- a/usafacts/delphi_usafacts/geo.py
+++ b/usafacts/delphi_usafacts/geo.py
@@ -25,37 +25,12 @@ REPLACE_FIPS = [
 ]
 
 
-FIPS_TO_STATE = {v: k.lower() for k, v in STATE_TO_FIPS.items()}
-
 # Valid geographical resolutions output by this indicator.
 VALID_GEO_RES = ("county", "state", "msa", "hrr")
 # Sensors that report proportions.  For geo resolutions with unallocated cases
 # or deaths, we avoid reporting these sensors.
 PROP_SENSORS = ("incidence", "cumulative_prop")
 
-def fips_to_state(fips: str) -> str:
-    """Wrapper that handles exceptions to the FIPS scheme in the USAFacts data.
-
-    All the county FIPS codes are mapped to state by taking the first two
-    digits of the five digit, zero-padded county FIPS and applying
-    FIPS_TO_STATE to map it to the two-letter postal abbreviation.
-
-    Parameters
-    ----------
-    fips: str
-        Five digit, zero padded county FIPS code
-
-    Returns
-    -------
-    str
-        Two-letter postal abbreviation, lower case.
-
-    Raises
-    ------
-    KeyError
-        Inputted FIPS code not recognized.
-    """
-    return FIPS_TO_STATE[fips[:2]]
 
 def disburse(df: pd.DataFrame, pooled_fips: str, fips_list: list):
     """Disburse counts from POOLED_FIPS equally to the counties in FIPS_LIST.
diff --git a/usafacts/delphi_usafacts/pull.py b/usafacts/delphi_usafacts/pull.py
index 045e65b3..7e5d8a82 100644
--- a/usafacts/delphi_usafacts/pull.py
+++ b/usafacts/delphi_usafacts/pull.py
@@ -73,10 +73,6 @@ def pull_usafacts_data(base_url: str, metric: str, geo_mapper: GeoMapper) -> pd.
         & (df["FIPS"] != 2270)
     ]
 
-    # Merge in population LOWERCASE, consistent across confirmed and deaths
-    # Population for unassigned cases/deaths is NAN
-    df = df.merge(pop_df, on="FIPS", how="left")
-
     # Change FIPS from 0 to XX000 for statewise unallocated cases/deaths
     unassigned_index = (df['FIPS'] == 0)
     df.loc[unassigned_index, "FIPS"] = df["stateFIPS"].loc[unassigned_index].values * 1000
Repository: cmu-delphi/covidcast-indicators
Issue: Indicator Runner can't find `run_module`

Stacktrace:

```
Traceback (most recent call last):
  File "/home/indicators/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/indicators/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/data2/indicators/runtime/quidel_covidtest/env/lib/python3.8/site-packages/delphi_utils/runner.py", line 49, in <module>
    run_indicator_pipeline(indicator_module.run_module, validator_from_params, archiver_from_params)
AttributeError: module 'delphi_quidel_covidtest' has no attribute 'run_module'
```

I've patched this in prod by adding 

```
from .run import run_module
```

to all the indicator `__init__.py` files so that the weekend schedule will be okay, but we should put in a more permanent fix before the next prod release.

Changed files:
_delphi_utils_python/delphi_utils/runner.py

diff --git a/_delphi_utils_python/delphi_utils/runner.py b/_delphi_utils_python/delphi_utils/runner.py
index 01f090b6..a6f1856f 100644
--- a/_delphi_utils_python/delphi_utils/runner.py
+++ b/_delphi_utils_python/delphi_utils/runner.py
@@ -43,7 +43,9 @@ if __name__ == "__main__":
     parser.add_argument("indicator_name",
                         type=str,
                         help="Name of the Python package containing the indicator.  This package "
-                             "must export a `run_module(params)` function.")
+                             "must export a `run.run_module(params)` function.")
     args = parser.parse_args()
     indicator_module = importlib.import_module(args.indicator_name)
-    run_indicator_pipeline(indicator_module.run_module, validator_from_params, archiver_from_params)
+    run_indicator_pipeline(indicator_module.run.run_module,
+                           validator_from_params,
+                           archiver_from_params)
Repository: cmu-delphi/covidcast-indicators
Issue: Archive differ run-as-module fails with AttributeError

**Actual Behavior:**

I am working on prototyping a multi-step indicator pipeline using the combo cases and deaths indicator. When I run the archive differ as a standalone module, it fails with an AttributeError, traceback below. It appears that the archive module main is improperly attempting to access dict keys as object attributes, since `cache_dir` is indeed present in the params read from disk:

```
$ env/bin/python -i -m delphi_utils.archive --archive_type s3 --indicator_prefix combo_cases_deaths
/usr/lib/python3.8/runpy.py:127: RuntimeWarning: 'delphi_utils.archive' found in sys.modules after import of package 'delphi_utils', but prior to execution of 'delphi_utils.archive'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/krivard/projects/covid/auto/covidcast-indicators/_delphi_utils_python/delphi_utils/archive.py", line 590, in <module>
    params.cache_dir,
AttributeError: 'dict' object has no attribute 'cache_dir'
>>> params
{'static_file_dir': './static', 'export_dir': './receiving', 'cache_dir': './cache', 'export_start_date': [2020, 4, 1], 'date_range': 'all', 'source': 'indicator-combination', 'wip_signal': [], 'aws_credentials': {'aws_access_key_id': '[elided]', 'aws_secret_access_key': '[elided]'}, 'bucket_name': 'delphi-covidcast-indicator-output-dev'}
```

**Expected behavior**

Either make `read_params()` return an Object, or make `archive.__main__` access params using dict-key notation.



Changed files:
_delphi_utils_python/delphi_utils/archive.py

diff --git a/_delphi_utils_python/delphi_utils/archive.py b/_delphi_utils_python/delphi_utils/archive.py
index 84f22f03..42f2ab4f 100644
--- a/_delphi_utils_python/delphi_utils/archive.py
+++ b/_delphi_utils_python/delphi_utils/archive.py
@@ -588,11 +588,11 @@ if __name__ == "__main__":
     args = parser.parse_args()
     params = read_params()
     run_module(args.archive_type,
-               params.cache_dir,
-               params.export_dir,
-               aws_credentials=params.aws_credentials,
+               params["cache_dir"],
+               params["export_dir"],
+               aws_credentials=params["aws_credentials"],
                branch_name=args.branch_name,
-               bucket_name=params.bucket_name,
+               bucket_name=params["bucket_name"],
                commit_message=args.commit_message,
                commit_partial_success=args.commit_partial_success,
                indicator_prefix=args.indicator_prefix,
Repository: cmu-delphi/covidcast-indicators
Issue: Fix doulbe/half prop problem in USAFacts this time

**Actual Behavior:**

Same bug as #325, in USAFacts this time

**Expected behavior**

State prop signals should incorporate 1x the state population (which requires ignoring the population of the megafips during county->state aggregation)


Changed files:
usafacts/tests/test_geo.py, usafacts/delphi_usafacts/geo.py

diff --git a/usafacts/delphi_usafacts/geo.py b/usafacts/delphi_usafacts/geo.py
index 53542a6c..5979a5e2 100644
--- a/usafacts/delphi_usafacts/geo.py
+++ b/usafacts/delphi_usafacts/geo.py
@@ -107,6 +107,13 @@ def geo_map(df: pd.DataFrame, geo_res: str, map_df: pd.DataFrame, sensor: str):
         df = df.append(unassigned_counties)
         geo_mapper = GeoMapper()
         df = geo_mapper.add_geocode(df, "fips", "state_id", new_col="geo_id")
+
+        # Zero out the state FIPS population to avoid double counting.
+        df = df.set_index("fips")
+        state_fips_codes = {str(x).zfill(2) + "000" for x in range(1, 73)}
+        subset_state_fips_codes = set(df.index.values) & state_fips_codes
+        df.loc[subset_state_fips_codes, "population"] = 0
+        df = df.reset_index()
     elif geo_res in ("msa", "hrr"):
         # Map "missing" secondary FIPS to those that are in our canonical set
         for fips, fips_list in SECONDARY_FIPS:
diff --git a/usafacts/tests/test_geo.py b/usafacts/tests/test_geo.py
index 33fe7dd3..c6f52a1a 100644
--- a/usafacts/tests/test_geo.py
+++ b/usafacts/tests/test_geo.py
@@ -76,11 +76,11 @@ class TestGeoMap:
         """Tests that values are correctly aggregated at the state level."""
         df = pd.DataFrame(
             {
-                "fips": ["04001", "04003", "04009", "25023"],
-                "timestamp": ["2020-02-15", "2020-02-15", "2020-02-15", "2020-02-15"],
-                "new_counts": [10, 15, 2, 13],
-                "cumulative_counts": [100, 20, 45, 60],
-                "population": [100, 2100, 300, 25],
+                "fips": ["04001", "04003", "04009", "25023", "25000"],
+                "timestamp": ["2020-02-15", "2020-02-15", "2020-02-15", "2020-02-15", "2020-02-15"],
+                "new_counts": [10, 15, 2, 13, 0],
+                "cumulative_counts": [100, 20, 45, 60, 0],
+                "population": [100, 2100, 300, 25, 25],
             }
         )
Repository: cmu-delphi/covidcast-indicators
Issue: USAFacts crashing; no longer includes nyc_dummy_row

Traceback from 5 & 6 February:
```
Traceback (most recent call last):
  File "/home/indicators/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/indicators/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/indicators/runtime/usafacts/delphi_usafacts/__main__.py", line 11, in <module>
    run_module()  # pragma: no cover
  File "/home/indicators/runtime/usafacts/delphi_usafacts/run.py", line 94, in run_module
    dfs = {metric: pull_usafacts_data(base_url, metric, geo_mapper) for metric in METRICS}
  File "/home/indicators/runtime/usafacts/delphi_usafacts/run.py", line 94, in <dictcomp>
    dfs = {metric: pull_usafacts_data(base_url, metric, geo_mapper) for metric in METRICS}
  File "/home/indicators/runtime/usafacts/delphi_usafacts/pull.py", line 90, in pull_usafacts_data
    assert len(nyc_dummy_row) == 1
AssertionError
```

Changed files:
usafacts/delphi_usafacts/pull.py

diff --git a/usafacts/delphi_usafacts/pull.py b/usafacts/delphi_usafacts/pull.py
index 6e074885..fec98fdf 100644
--- a/usafacts/delphi_usafacts/pull.py
+++ b/usafacts/delphi_usafacts/pull.py
@@ -87,7 +87,6 @@ def pull_usafacts_data(base_url: str, metric: str, geo_mapper: GeoMapper) -> pd.
     # in the call to `add_population_column()`.  We pull it out here to
     # reinsert it after the population data is added.
     nyc_dummy_row = df[df["fips"] == "00001"]
-    assert len(nyc_dummy_row) == 1
 
     # Merge in population LOWERCASE, consistent across confirmed and deaths
     # Population for unassigned cases/deaths is NAN
Repository: cmu-delphi/covidcast-indicators
Issue: USAFacts should group by geo before smoothing

USAFacts smoothing just jams the whole data series in there without regard for geo_id:

https://github.com/cmu-delphi/covidcast-indicators/blob/e8035936647065e8b082b75a76d39046e99d7f81/usafacts/delphi_usafacts/run.py#L100-L101

& has been that way since November: https://github.com/cmu-delphi/covidcast-indicators/pull/429/files

The equivalent operation in JHU groups by what I'm assuming is geo_id first before applying the smoother in a transform operation:

https://github.com/cmu-delphi/covidcast-indicators/blob/e8035936647065e8b082b75a76d39046e99d7f81/jhu/delphi_jhu/run.py#L106

The grouping implementation is correct.

Changed files:
usafacts/delphi_usafacts/run.py, usafacts/tests/test_run.py

diff --git a/usafacts/delphi_usafacts/run.py b/usafacts/delphi_usafacts/run.py
index 96dc81c3..309f7e2c 100644
--- a/usafacts/delphi_usafacts/run.py
+++ b/usafacts/delphi_usafacts/run.py
@@ -98,7 +98,9 @@ def run_module():
         df = dfs[metric]
         # Aggregate to appropriate geographic resolution
         df = geo_map(df, geo_res, sensor)
-        df["val"] = SMOOTHERS_MAP[smoother][0].smooth(df[sensor].values)
+        df["val"] = df[["geo_id", sensor]].groupby("geo_id")[sensor].transform(
+            SMOOTHERS_MAP[smoother][0].smooth
+        )
         df["se"] = np.nan
         df["sample_size"] = np.nan
         # Drop early entries where data insufficient for smoothing
diff --git a/usafacts/tests/test_run.py b/usafacts/tests/test_run.py
index 2b8bfab9..2134c444 100644
--- a/usafacts/tests/test_run.py
+++ b/usafacts/tests/test_run.py
@@ -39,6 +39,8 @@ class TestRun:
         for date in dates:
             for geo in geos:
                 for metric in metrics:
+                    if "7dav" in metric and date in dates[:6]:
+                        continue  # there are no 7dav signals for first 6 days
                     expected_files += [date + "_" + geo + "_" + metric + ".csv"]
 
         assert set(csv_files) == set(expected_files)
Repository: hedera-lang-learn/hedera
Issue: Long load times and 503 errors in dev

When loading the dev instance, some page loads are taking longer than expected, and others are throwing a 503 error.

Changed files:
hedera/api.py

diff --git a/hedera/api.py b/hedera/api.py
index ec6c160..bb3d4f2 100644
--- a/hedera/api.py
+++ b/hedera/api.py
@@ -549,23 +549,14 @@ def get_any_vocablist_by_id(user_id, lang, vocablist_id):
 
 
 class PartialMatchFormLookupAPI(APIView):
+
     def get_data(self):
         form = self.kwargs.get("form")
         lang = self.kwargs.get("lang")
-        # gets list of forms - Match exactly since startswith doesnt match exactly
-        forms = FormToLemma.objects.filter(lang=lang, form=form)
-        if not forms:
-            forms = FormToLemma.objects.filter(lang=lang, form__startswith=form)
-            if not forms:
-                forms = FormToLemma.objects.filter(lang=lang, form__startswith=form.lower())
-        lemma_list = [form.get_lemma() for form in forms]
-        lemma_dict = {}
-        sorted_lemma_list = (sorted(lemma_list, key=lambda i: i["rank"]))
-        distinct_lemma_list = []
-        # removing duplicates using a dict lookup
-        for lemma_obj in sorted_lemma_list:
-            lemma_word = lemma_obj["lemma"]
-            if lemma_word not in lemma_dict:
-                lemma_dict[lemma_word] = lemma_obj
-                distinct_lemma_list.append(lemma_obj)
-        return distinct_lemma_list
+        lemmas = Lemma.objects.filter(lang=lang, forms__form=form).distinct().order_by("rank").prefetch_related("glosses")
+        if not lemmas:
+            lemmas = Lemma.objects.filter(lang=lang, forms__form__startswith=form).distinct().order_by("rank").prefetch_related("glosses")
+            if not lemmas:
+                lemmas = Lemma.objects.filter(lang=lang, forms__form__startswith=form.lower()).distinct().order_by("rank").prefetch_related("glosses")
+        lemma_list = [lemma.to_dict() for lemma in lemmas]
+        return lemma_list