anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: 2dos/dk64-randomizer
Issue: Impossible Beat K. Rool seed via character lockout on 2.0 release.

I need Chunky to fight K. Rool so, Chunky is in Jungle Japes Mini-Monkey cage, but Tiny is in a shop for Chunky to purchase meaning no way beyond King Kutout to obtain Chunky or Tiny which I would expect not to be in logic despite the patch that prevented smuggling Kongs from Kutout before being removed. (or it now fails to prevent smuggling Kongs?)
If that is in glitched logic then I think it should have a setting toggle for it.

Here is my settings string in a txt and spoiler log file (zipped) attached.
[DK64r 133769 Settings.txt](https://github.com/2dos/DK64-Randomizer/files/11973766/DK64r.133769.Settings.txt)
[133769-spoilerlog.zip](https://github.com/2dos/DK64-Randomizer/files/11973771/133769-spoilerlog.zip)


Changed files:
randomizer/LogicFiles/JungleJapes.py

diff --git a/randomizer/LogicFiles/JungleJapes.py b/randomizer/LogicFiles/JungleJapes.py
index 172fd713..3de1fcb9 100644
--- a/randomizer/LogicFiles/JungleJapes.py
+++ b/randomizer/LogicFiles/JungleJapes.py
@@ -103,7 +103,7 @@ LogicRegions = {
     ]),
 
     Regions.JapesBeyondFeatherGate: Region("Japes Beyond Feather Gate", "Hive Tunnel Area", Levels.JungleJapes, True, None, [
-        LocationLogic(Locations.JapesTinyStump, lambda l: (l.mini and l.istiny) or l.phasewalk or l.CanSkew(True) or l.CanSkew(False)),
+        LocationLogic(Locations.JapesTinyStump, lambda l: (((l.mini and l.istiny) or l.phasewalk or l.CanSkew(True) or l.CanSkew(False)) and (l.istiny or l.settings.free_trade_items))),
         LocationLogic(Locations.JapesChunkyGiantBonusBarrel, lambda l: l.hunkyChunky and l.ischunky, MinigameType.BonusBarrel),
     ], [
         Event(Events.JapesW5aTagged, lambda l: True),
Repository: virtool/virtool
Issue: InvalidStateError: invalid state

Sentry Issue: [SERVER-RE](https://sentry.io/cph/server/issues/899628324/?referrer=github_integration)

```
InvalidStateError: invalid state
  File "uvloop/cbhandles.pyx", line 66, in uvloop.loop.Handle._run
  File "usr/local/lib/python3.6/site-packages/motor/core.py", line 1244, in _on_get_more

Exception in callback <bound method AgnosticLatentCommandCursor._on_get_more of AsyncIOMotorLatentCommandCursor(<pymongo.command_cursor.CommandCursor object at 0x7fa1cedac668>)>
handle: <Handle AgnosticLatentCommandCursor._on_get_more>
```

Changed files:
virtool/api/references.py

diff --git a/virtool/api/references.py b/virtool/api/references.py
index 330a41452..481950174 100644
--- a/virtool/api/references.py
+++ b/virtool/api/references.py
@@ -101,9 +101,14 @@ async def get(req):
     except (KeyError, TypeError):
         internal_control_id = None
 
-    document.update(await virtool.db.references.get_computed(db, ref_id, internal_control_id))
+    computed = await asyncio.shield(virtool.db.references.get_computed(db, ref_id, internal_control_id))
+
+    users = await asyncio.shield(virtool.db.users.attach_identicons(db, document["users"]))
 
-    document["users"] = await virtool.db.users.attach_identicons(db, document["users"])
+    document.update({
+        **computed,
+        "users": users
+    })
 
     return json_response(virtool.utils.base_processor(document))
Repository: virtool/virtool
Issue: KeyError: "Key not found: 'X-RateLimit-Remaining'"

Sentry Issue: [SERVER-PY](https://sentry.io/cph/server/issues/845615284/?referrer=github_integration)

```
KeyError: "Key not found: 'X-RateLimit-Remaining'"
  File "drone/src/virtool/db/hmm.py", line 307, in refresh
  File "drone/src/virtool/db/hmm.py", line 95, in fetch_and_update_release
  File "drone/src/virtool/github.py", line 81, in get_release
  File "multidict/_multidict.pyx", line 140, in multidict._multidict._Base.__getitem__
  File "multidict/_multidict.pyx", line 135, in multidict._multidict._Base._getone

Job processing failed
job: <Job coro=<<coroutine object refresh at 0x7f30dccc2468>>>
```

Changed files:
virtool/github.py

diff --git a/virtool/github.py b/virtool/github.py
index d95a62106..7308801bd 100644
--- a/virtool/github.py
+++ b/virtool/github.py
@@ -74,13 +74,10 @@ async def get_release(settings, session, slug, etag=None, release_id="latest"):
         headers["If-None-Match"] = etag
 
     async with virtool.http.proxy.ProxyRequest(settings, session.get, url, headers=headers) as resp:
-        logger.debug("Fetched release: {}/{} ({} - {}/{})".format(
-            slug,
-            release_id,
-            resp.status,
-            resp.headers["X-RateLimit-Remaining"],
-            resp.headers["X-RateLimit-Limit"]
-        ))
+        rate_limit_remaining = resp.headers.get("X-RateLimit-Remaining", "00")
+        rate_limit = resp.headers.get("X-RateLimit-Limit", "00")
+
+        logger.debug(f"Fetched release: {slug}/{release_id} ({resp.status} - {rate_limit_remaining}/{rate_limit})")
 
         if resp.status == 200:
             data = await resp.json()
Repository: virtool/virtool
Issue: Don't return API errors for unknown search parameter keys

This is not a failure condition. Errors will still be returned for invalid search parameters values.

Changed files:
virtool/handlers/samples.py, tests/handlers/test_samples.py

diff --git a/tests/handlers/test_samples.py b/tests/handlers/test_samples.py
index 48b5bc260..aad6a8f4e 100644
--- a/tests/handlers/test_samples.py
+++ b/tests/handlers/test_samples.py
@@ -178,12 +178,15 @@ class TestFind:
     async def test_invalid_query(self, spawn_client, resp_is):
         client = await spawn_client()
 
-        resp = await client.get("/api/samples?per_pag=12")
+        resp = await client.get("/api/samples?per_page=five")
 
         assert resp.status == 422
 
         assert await resp_is.invalid_query(resp, {
-            "per_pag": ["unknown field"]
+                'per_page': [
+                    "field 'per_page' cannot be coerced: invalid literal for int() with base 10: 'five'",
+                    'must be of integer type'
+                ]
         })
 
 
diff --git a/virtool/handlers/samples.py b/virtool/handlers/samples.py
index 5681e8fec..5bf0dec00 100644
--- a/virtool/handlers/samples.py
+++ b/virtool/handlers/samples.py
@@ -21,7 +21,7 @@ async def find(req):
         "find": {"type": "string", "default": "", "coerce": str},
         "page": {"type": "integer", "coerce": int, "default": 1, "min": 1},
         "per_page": {"type": "integer", "coerce": int, "default": 15, "min": 1, "max": 100}
-    })
+    }, allow_unknown=True)
 
     if not v(dict(req.query)):
         return invalid_query(v.errors)
Repository: virtool/virtool
Issue: Changing groups permissions does not immediately affect members

Permissions can be applied to members by toggling off and on their membership in the changed group.

Changed files:
virtool/handlers/groups.py

diff --git a/virtool/handlers/groups.py b/virtool/handlers/groups.py
index 98d1c3786..daa62a4ff 100644
--- a/virtool/handlers/groups.py
+++ b/virtool/handlers/groups.py
@@ -92,6 +92,8 @@ async def update_permissions(req):
         }
     }, return_document=ReturnDocument.AFTER)
 
+    await virtool.user_groups.update_member_users(db, group_id)
+
     return json_response(virtool.utils.base_processor(document))
Repository: virtool/virtool
Issue: Strange behavior when updating the CPU and memory Limits

From #474:

> I get strange behavior when updating the CPU and Memory Limits. Individually they update fine, but when updating one after the other the least recent change seems to get overwritten by the default values of proc = 8 and mem = 16.

> Changing multiple fields in task-specific limits also overwrites changes with default values.
memory limit accepts float values (one decimal place) but will only save the floored integer equivalent.

Changed files:
virtool/handlers/settings.py

diff --git a/virtool/handlers/settings.py b/virtool/handlers/settings.py
index 2e2331a7b..8c2c85aea 100644
--- a/virtool/handlers/settings.py
+++ b/virtool/handlers/settings.py
@@ -23,19 +23,23 @@ async def get_one(req):
 
 
 @protected("modify_settings")
-@validation(virtool.app_settings.SCHEMA)
 async def update(req):
     """
     Update application settings based on request data.
 
     """
-    data = req["data"]
+    data = await req.json()
 
     keys = data.keys()
 
+    v = Validator(virtool.app_settings.SCHEMA)
+
+    if not v(data):
+        return invalid_input(v.errors)
+
     settings = req.app["settings"]
 
-    settings.data.update({key: data[key] for key in keys})
+    settings.data.update({key: v.document[key] for key in keys})
 
     await settings.write()
Repository: virtool/virtool
Issue: AttributeError: module 'virtool.samples.db' has no attribute 'get_sample_path'

Error when running analyses of legacy samples:

```
Traceback (most recent call last):
File "/drone/src/virtool/jobs/job.py", line 166, in run
File "/drone/src/virtool/jobs/analysis.py", line 136, in prepare_reads
File "/drone/src/virtool/samples/utils.py", line 97, in join_legacy_read_paths
AttributeError: module 'virtool.samples.db' has no attribute 'get_sample_path'
```

Changed files:
virtool/hmm/api.py, virtool/jobs/api.py, virtool/files/api.py, virtool/hmm/db.py, virtool/samples/utils.py

diff --git a/virtool/files/api.py b/virtool/files/api.py
index 172840d26..0b2bd6e82 100644
--- a/virtool/files/api.py
+++ b/virtool/files/api.py
@@ -2,7 +2,6 @@
 Provides request handlers for managing and viewing files.
 
 """
-import os
 import pymongo
 
 import virtool.files.db
diff --git a/virtool/hmm/api.py b/virtool/hmm/api.py
index b0e35e21d..4f1c3a1bf 100644
--- a/virtool/hmm/api.py
+++ b/virtool/hmm/api.py
@@ -6,7 +6,6 @@ import aiojobs.aiohttp
 import virtool.db.utils
 import virtool.errors
 import virtool.github
-import virtool.hmm
 import virtool.hmm.db
 import virtool.http.routes
 import virtool.processes.db
diff --git a/virtool/hmm/db.py b/virtool/hmm/db.py
index f86678077..4108e9eb3 100644
--- a/virtool/hmm/db.py
+++ b/virtool/hmm/db.py
@@ -1,5 +1,3 @@
-import aiofiles
-import aiohttp.web
 import asyncio
 import json
 import logging
@@ -8,10 +6,9 @@ import shutil
 
 import aiofiles
 import aiohttp.client_exceptions
-import semver
+import aiohttp.web
 
 import virtool.analyses.utils
-
 import virtool.db.core
 import virtool.db.utils
 import virtool.errors
diff --git a/virtool/jobs/api.py b/virtool/jobs/api.py
index 2137d3251..67f28b175 100644
--- a/virtool/jobs/api.py
+++ b/virtool/jobs/api.py
@@ -1,9 +1,9 @@
 import os
 
-import virtool.jobs.db
 import virtool.http.routes
-import virtool.jobs
+import virtool.jobs.db
 import virtool.resources
+import virtool.users.db
 import virtool.utils
 from virtool.api import compose_regex_query, conflict, json_response, no_content, not_found, paginate
 
@@ -34,6 +34,7 @@ async def find(req):
     )
 
     data["documents"].sort(key=lambda d: d["created_at"])
+    data["documents"] = await virtool.users.db.attach_identicons_to_documents(db, data["documents"])
 
     return json_response(data)
 
diff --git a/virtool/samples/utils.py b/virtool/samples/utils.py
index 78c2b535b..1dec67fb3 100644
--- a/virtool/samples/utils.py
+++ b/virtool/samples/utils.py
@@ -1,7 +1,5 @@
 import os
 
-import virtool.samples
-
 PATHOSCOPE_TASK_NAMES = [
     "pathoscope_bowtie",
     "pathoscope_barracuda"
@@ -94,7 +92,7 @@ def join_legacy_read_paths(settings: dict, sample):
     :return: a list of sample read paths
 
     """
-    sample_path = virtool.samples.db.get_sample_path(settings, sample["_id"])
+    sample_path = join_sample_path(settings, sample["_id"])
 
     if not all(f["raw"] for f in sample["files"]):
         if sample["paired"]:
Repository: virtool/virtool
Issue: Validate host resource and job limits on start

The server should exit if host resource and job limits are invalid.

- reuse code from older settings API handlers
- log fatal and `sys.exit(1)`

Changed files:
virtool/config.py

diff --git a/virtool/config.py b/virtool/config.py
index 667e4f8fa..3dc5bcad0 100644
--- a/virtool/config.py
+++ b/virtool/config.py
@@ -2,6 +2,7 @@ import argparse
 import json
 import logging
 import os
+import psutil
 import sys
 import urllib.parse
 from typing import Union
@@ -507,7 +508,11 @@ def resolve() -> dict:
 
     from_defaults = get_defaults()
 
-    return {**from_defaults, **from_env, **from_file, **from_args}
+    resolved = {**from_defaults, **from_env, **from_file, **from_args}
+
+    validate_limits(resolved)
+
+    return resolved
 
 
 def should_do_setup(config):
@@ -520,18 +525,39 @@ def should_do_setup(config):
     return not file_exists()
 
 
-def validate(data: dict) -> dict:
-    v = cerberus.Validator(SCHEMA, purge_unknown=True)
+def validate_limits(config):
+    cpu_count = psutil.cpu_count()
+    mem_total = psutil.virtual_memory().total
+
+    proc = int(config["proc"])
+    mem = int(config["mem"])
+
+    fatal = False
+
+    if proc > cpu_count:
+        fatal = True
+        logger.fatal(f"Configured proc limit ({proc}) exceeds host CPU count ({cpu_count})")
+
+    in_bytes = mem * 1024 * 1024 * 1024
+
+    if in_bytes > mem_total:
+        fatal = True
+        logger.fatal(f"Configured mem limit ({in_bytes}) exceeds host memory ({mem_total})")
+
+    for job_limit_key in JOB_LIMIT_KEYS:
+        resource_key = job_limit_key.split("_")[1]
 
-    if not v.validate(data):
-        logger.critical("Could not validate settings file", v.errors)
+        job_limit = int(config[job_limit_key])
+        host_limit = int(config[resource_key])
 
-        for error in v.errors:
-            logger.critical(f"\\t{error}")
+        if job_limit > host_limit:
+            fatal = True
+            logger.fatal(f"Configured {job_limit_key} ({job_limit}) exceeds instance {resource_key} limit ({host_limit})")
 
+    if fatal:
         sys.exit(1)
 
-    return v.document
+    return cpu_count, mem_total
 
 
 def write_to_file(data):
Repository: virtool/virtool
Issue: Virtool setup html template

I wasn't sure if this was expected behaviour or not:

Clicking on the "Save" button more than once for the Username and Password form will result in not being able to log in with those credentials.

Inspecting the POST body shows that the password gets replaced with "dummy password".

Changed files:
virtool/setup.py

diff --git a/virtool/setup.py b/virtool/setup.py
index 1f6a9fee6..2fc11d320 100644
--- a/virtool/setup.py
+++ b/virtool/setup.py
@@ -214,19 +214,20 @@ async def setup_db(req):
 async def setup_user(req):
     data = await req.post()
 
-    v = Validator({
-        "user_id": {"type": "string", "required": True},
-        "password": {"type": "string", "required": True}
-    }, allow_unknown=False)
+    if data["password"] != "dummy_password":
+        v = Validator({
+            "user_id": {"type": "string", "required": True},
+            "password": {"type": "string", "required": True}
+        }, allow_unknown=False)
 
-    v.validate(dict(data))
+        v.validate(dict(data))
 
-    data = v.document
+        data = v.document
 
-    req.app["setup"].update({
-        "first_user_id": data["user_id"],
-        "first_user_password": virtool.users.hash_password(data["password"])
-    })
+        req.app["setup"].update({
+            "first_user_id": data["user_id"],
+            "first_user_password": virtool.users.hash_password(data["password"])
+        })
 
     return web.HTTPFound("/setup")
Repository: virtool/virtool
Issue: DuplicateKeyError: E11000 duplicate key error collection: virtool.indexes index: version_1 dup key: { : 0 }

https://sentry.io/cph/server/issues/562788089/

```
DuplicateKeyError: E11000 duplicate key error collection: virtool.indexes index: version_1 dup key: { : 0 }
(10 additional frame(s) were not displayed)
...
  File "pymongo/collection.py", line 682, in insert_one
    session=session),
  File "pymongo/collection.py", line 598, in _insert
    bypass_doc_val, session)
  File "pymongo/collection.py", line 579, in _insert_one
    _check_write_command_response(result)
  File "pymongo/helpers.py", line 208, in _check_write_command_response
    _raise_last_write_error(write_errors)
  File "pymongo/helpers.py", line 189, in _raise_last_write_error
    raise DuplicateKeyError(error.get("errmsg"), 11000, error)

Error handling request
```

Results in an a `500` error on index build request (#690).

Changed files:
virtool/app.py

diff --git a/virtool/app.py b/virtool/app.py
index 33f252936..b6d213165 100644
--- a/virtool/app.py
+++ b/virtool/app.py
@@ -142,7 +142,8 @@ async def init_check_db(app):
     await db.history.create_index("otu.id")
     await db.history.create_index("index.id")
     await db.history.create_index("created_at")
-    await db.indexes.create_index("version", unique=True)
+    await db.history.drop_indexes()
+    await db.indexes.create_index([("version", 1), ("reference.id", 1)], unique=True)
     await db.keys.create_index("id", unique=True)
     await db.keys.create_index("user.id")
     await db.samples.create_index([("created_at", pymongo.DESCENDING)])
Repository: virtool/virtool
Issue: Analysis files not removed reliably on record deletion

When analyses are removed by user action, the associated files are not reliably removed.

Changed files:
virtool/api/analyses.py, tests/api/test_analyses.py

diff --git a/tests/api/test_analyses.py b/tests/api/test_analyses.py
index 7571563d8..1eba24d9f 100644
--- a/tests/api/test_analyses.py
+++ b/tests/api/test_analyses.py
@@ -86,10 +86,12 @@ async def test_get(ready, error, mocker, spawn_client, resp_is):
 
 
 @pytest.mark.parametrize("error", [None, "400", "403", "404", "409"])
-async def test_remove(error, spawn_client, resp_is):
+async def test_remove(mocker, error, spawn_client, resp_is):
 
     client = await spawn_client(authorize=True)
 
+    client.app["settings"]["data_path"] = "data"
+
     if error != "400":
         await client.db.samples.insert_one({
             "_id": "baz",
@@ -116,6 +118,8 @@ async def test_remove(error, spawn_client, resp_is):
             }
         })
 
+    m_remove =  mocker.patch("virtool.utils.rm")
+
     resp = await client.delete("/api/analyses/foobar")
 
     if error == "400":
@@ -138,6 +142,8 @@ async def test_remove(error, spawn_client, resp_is):
 
     assert await client.db.analyses.find_one() is None
 
+    assert m_remove.called_with("data/samples/baz/analyses/foobar", True)
+
 
 @pytest.mark.parametrize("error", [None, "400", "403", "404_analysis", "404_sequence", "409_algorithm", "409_ready"])
 async def test_blast(error, mocker, spawn_client, resp_is, static_time):
diff --git a/virtool/api/analyses.py b/virtool/api/analyses.py
index aab632321..9e8ba07f2 100644
--- a/virtool/api/analyses.py
+++ b/virtool/api/analyses.py
@@ -2,6 +2,7 @@
 Provides request handlers for managing and viewing analyses.
 
 """
+import os
 import aiojobs.aiohttp
 
 import virtool.analyses
@@ -80,6 +81,10 @@ async def remove(req):
 
     await db.analyses.delete_one({"_id": analysis_id})
 
+    path = os.path.join(req.app["settings"]["data_path"], "samples", sample_id, "analysis", analysis_id)
+
+    await req.app["run_in_thread"](virtool.utils.rm, path, True)
+
     await virtool.db.samples.recalculate_algorithm_tags(db, sample_id)
 
     return no_content()
Repository: virtool/virtool
Issue: Some analyses incorrectly show spinners

This issue is probably related to sample rights.

Spinners are shown for some users, but not others when their rights on the samples are different.

Changed files:
virtool/jobs/nuvs.py, virtool/jobs/pathoscope.py

diff --git a/virtool/jobs/nuvs.py b/virtool/jobs/nuvs.py
index e4fc8c98d..c36c23888 100644
--- a/virtool/jobs/nuvs.py
+++ b/virtool/jobs/nuvs.py
@@ -393,11 +393,20 @@ class Job(virtool.jobs.job.Job):
         self.dispatch("samples", "update", [sample_id])
 
     def cleanup(self):
+        self.db.analyses.delete_one({"_id": self.params["analysis_id"]})
+
         try:
             self.temp_dir.cleanup()
         except AttributeError:
             pass
 
-        virtool.db.sync.recalculate_algorithm_tags(self.db, self.params["sample_id"])
+        try:
+            shutil.rmtree(self.params["analysis_path"])
+        except FileNotFoundError:
+            pass
+
+        sample_id = self.params["sample_id"]
+
+        virtool.db.sync.recalculate_algorithm_tags(self.db, sample_id)
 
-        super().cleanup()
+        self.dispatch("samples", "update", [sample_id])
diff --git a/virtool/jobs/pathoscope.py b/virtool/jobs/pathoscope.py
index 0dfabbf0d..6ac9784b4 100644
--- a/virtool/jobs/pathoscope.py
+++ b/virtool/jobs/pathoscope.py
@@ -439,7 +439,11 @@ class Job(virtool.jobs.job.Job):
         except FileNotFoundError:
             pass
 
-        virtool.db.sync.recalculate_algorithm_tags(self.db, self.params["sample_id"])
+        sample_id = self.params["sample_id"]
+
+        virtool.db.sync.recalculate_algorithm_tags(self.db, sample_id)
+
+        self.dispatch("samples", "update", [sample_id])
 
     def cleanup_indexes(self):
         pass
Repository: virtool/virtool
Issue: Get data type and organism reference metadata from remote

Installing a remote reference currently does not set the `data_type` and `organism` fields of the local copy.

- [x] fix this issue
- [x] ensure cloned and imported references also support this properly
- [ ] disallow changing `data_type` and `organism` during creation of imported references

Changed files:
virtool/db/references.py

diff --git a/virtool/db/references.py b/virtool/db/references.py
index af3bd1995..67935e6d4 100644
--- a/virtool/db/references.py
+++ b/virtool/db/references.py
@@ -894,6 +894,22 @@ async def finish_remote(app, release, ref_id, created_at, process_id, user_id):
             errors=["Could not download reference data"]
         )
 
+    try:
+        data_type = import_data["data_type"]
+    except KeyError:
+        return await virtool.db.processes.update(
+            db,
+            process_id,
+            errors=["Could not infer data type"]
+        )
+
+    await db.references.update_one({"_id": ref_id}, {
+        "$set": {
+            "data_type": data_type,
+            "organism": import_data.get("organism", "Unknown")
+        }
+    })
+
     errors = virtool.references.check_import_data(
         import_data,
         strict=True,
Repository: virtool/virtool
Issue: Edit Sequence updates with empty fields

Unlike add-sequence, edit-sequence will update with empty definition or sequence fields without error. 

Changed files:
virtool/api/otus.py

diff --git a/virtool/api/otus.py b/virtool/api/otus.py
index 961f8b0fa..5201465be 100644
--- a/virtool/api/otus.py
+++ b/virtool/api/otus.py
@@ -685,11 +685,24 @@ async def create_sequence(req):
 
 
 @routes.patch("/api/otus/{otu_id}/isolates/{isolate_id}/sequences/{sequence_id}", schema={
-    "host": {"type": "string"},
-    "definition": {"type": "string"},
-    "segment": {"type": "string"},
-    "sequence": {"type": "string"},
-    "schema": {"type": "list"}
+    "accession": {
+        "type": "string",
+        "empty": False
+    },
+    "host": {
+        "type": "string"
+    },
+    "definition": {
+        "type": "string",
+        "empty": False
+    },
+    "segment": {
+        "type": "string"
+    },
+    "sequence": {
+        "type": "string",
+        "empty": False
+    }
 })
 async def edit_sequence(req):
     db, data = req.app["db"], req["data"]
Repository: virtool/virtool
Issue: The cloned_from field is not included in ref find response

> Cloned references seem to be missing the cloned_from field and information within their state.references.documents entry.

Changed files:
virtool/db/references.py

diff --git a/virtool/db/references.py b/virtool/db/references.py
index 3575e9b4f..3d6369b27 100644
--- a/virtool/db/references.py
+++ b/virtool/db/references.py
@@ -18,6 +18,7 @@ import virtool.utils
 PROJECTION = [
     "_id",
     "remotes_from",
+    "cloned_from",
     "created_at",
     "data_type",
     "imported_from",
Repository: virtool/virtool
Issue: Long-lived sessions expiring early

Long-lived (Remember Me) sessions are only lasting for 30 minutes instead of the expected 30 days.

Changed files:
virtool/http/auth.py

diff --git a/virtool/http/auth.py b/virtool/http/auth.py
index e941b32a5..c299ff765 100644
--- a/virtool/http/auth.py
+++ b/virtool/http/auth.py
@@ -57,12 +57,12 @@ class Client:
         self.session_id = session_id
 
 
-async def auth_response(req, return_to, user_id):
+async def auth_response(req, return_to, user_id, remember):
     db = req.app["db"]
     client = req["client"]
 
     # Create a new authenticated session for the user. This is identical to the process followed for logging in.
-    session, token = await virtool.db.sessions.replace_session(db, client.session_id, get_ip(req), user_id)
+    session, token = await virtool.db.sessions.replace_session(db, client.session_id, get_ip(req), user_id, remember)
 
     req["client"].authorize(session, False)
     req["client"].session_id = session["_id"]
@@ -335,7 +335,7 @@ async def login_post_handler(req: web.Request) -> web.Response:
         reset_code = await virtool.db.sessions.get_reset_code(db, client.session_id, user_id)
         return web.Response(status=302, headers={"Location": f"/reset?return_to={return_to}&code={reset_code}"})
 
-    return await auth_response(req, return_to, user_id)
+    return await auth_response(req, return_to, user_id, remember)
 
 
 async def reset_get_handler(req: web.Request) -> web.Response:
@@ -444,4 +444,4 @@ async def reset_post_handler(req: web.Request) -> web.Response:
 
     # Authenticate and return a redirect response to the `return_to` path. This is identical to the process used for
     # successful login requests.
-    return await auth_response(req, return_to, user_id)
+    return await auth_response(req, return_to, user_id, False)
Repository: virtool/virtool
Issue: Bypass API permission check if administrator

Currently `403 Forbidden` is returned if the client lacks a required permission even if they are an administrator.

Administrator status should always trump permissions.

Changed files:
virtool/http/routes.py

diff --git a/virtool/http/routes.py b/virtool/http/routes.py
index cff3dc208..657acd2b4 100644
--- a/virtool/http/routes.py
+++ b/virtool/http/routes.py
@@ -47,17 +47,18 @@ def protect(route_decorator, admin, permission, public, schema):
                     "message": "Requires authorization"
                 }, status=401)
 
-            if admin and not req["client"].administrator:
-                return json_response({
-                    "id": "not_permitted",
-                    "message": "Requires administrative privilege"
-                }, status=403)
-
-            if permission and not req["client"].permissions[permission]:
-                return json_response({
-                    "id": "not_permitted",
-                    "message": "Not permitted"
-                }, status=403)
+            if not req["client"].administrator:
+                if admin:
+                    return json_response({
+                        "id": "not_permitted",
+                        "message": "Requires administrative privilege"
+                    }, status=403)
+
+                if permission and not req["client"].permissions[permission]:
+                    return json_response({
+                        "id": "not_permitted",
+                        "message": "Not permitted"
+                    }, status=403)
 
             content_type = req.headers.get("Content-type", "")
Repository: virtool/virtool
Issue: User primary group deselect

Should admin be able to unset primary groups for users? 

Currently attempting to select the "none" option for a user already assigned a primary group results in an `PATCH http://localhost:9950/api/users/a1 400 (Bad Request)` error with a `{ "id": "bad_request", "message": "Primary group does not exist"}` response.

Changed files:
virtool/api/users.py

diff --git a/virtool/api/users.py b/virtool/api/users.py
index 6195bc693..c791e1b46 100644
--- a/virtool/api/users.py
+++ b/virtool/api/users.py
@@ -132,7 +132,9 @@ async def edit(req):
         if missing:
             return bad_request("Groups do not exist: " + ", ".join(missing))
 
-    if "primary_group" in data and data["primary_group"] not in groups:
+    primary_group = data.get("primary_group", None)
+
+    if primary_group and primary_group not in groups:
         return bad_request("Primary group does not exist")
 
     user_id = req.match_info["user_id"]
Repository: virtool/virtool
Issue: Host subtraction mapping percentage - specific example

Sample Q18B has more host reads mapped than the total number of reads. Wondering how pathoscope determined this mapping percentage?

Changed files:
virtool/pathoscope.py

diff --git a/virtool/pathoscope.py b/virtool/pathoscope.py
index 8cbefa595..7ca275ff5 100755
--- a/virtool/pathoscope.py
+++ b/virtool/pathoscope.py
@@ -476,21 +476,20 @@ def subtract(analysis_path, host_scores):
 
     out_path = os.path.join(analysis_path, "subtracted.vta")
 
-    total_count = 0
+    subtracted_read_ids = set()
 
     with open(vta_path, "r") as vta_handle:
         with open(out_path, "w") as out_handle:
             for line in vta_handle:
-                total_count += 1
                 fields = line.rstrip().split(",")
                 read_id = fields[0]
                 if isolates_high_scores[read_id] > host_scores.get(read_id, 0):
                     out_handle.write(line)
                 else:
-                    subtracted_count += 1
+                    subtracted_read_ids.add(read_id)
 
     os.remove(vta_path)
 
     shutil.move(out_path, vta_path)
 
-    return subtracted_count
+    return len(subtracted_read_ids)
Repository: virtool/virtool
Issue: Get unbuilt changes 404

- building a new index results in `GET http://localhost:9950/api/indexes/unbuilt 404 (Not Found)`
(same for `/api/refs/:refId/indexes/unbuilt`)

Changed files:
virtool/api/references.py, virtool/db/history.py

diff --git a/virtool/api/references.py b/virtool/api/references.py
index 1d73b046c..1b1b3b743 100644
--- a/virtool/api/references.py
+++ b/virtool/api/references.py
@@ -95,6 +95,16 @@ async def find_history(req):
         "reference.id": ref_id
     }
 
+    unbuilt = req.query.get("unbuilt", None)
+
+    if unbuilt == "true":
+        base_query["index.id"] = "unbuilt"
+
+    elif unbuilt == "false":
+        base_query["index.id"] = {
+            "$ne": "unbuilt"
+        }
+
     data = await virtool.db.history.find(
         db,
         req.query,
@@ -196,6 +206,7 @@ async def create(req):
 
         document = await virtool.db.references.create_for_import(
             db,
+            settings,
             data["name"],
             data["description"],
             data["public"],
@@ -300,26 +311,6 @@ async def edit(req):
     return json_response(document)
 
 
-@routes.get("/api/refs/{ref_id}/unbuilt")
-async def get_unbuilt_changes(req):
-    """
-    Get a JSON document describing the unbuilt changes that could be used to create a new index.
-
-    """
-    db = req.app["db"]
-
-    ref_id = req.match_info["ref_id"]
-
-    history = await db.history.find({
-        "reference.id": ref_id,
-        "index.id": "unbuilt"
-    }, virtool.db.history.LIST_PROJECTION).to_list(None)
-
-    return json_response({
-        "history": [virtool.utils.base_processor(c) for c in history]
-    })
-
-
 @routes.delete("/api/refs/{ref_id}")
 async def remove(req):
     """
diff --git a/virtool/db/history.py b/virtool/db/history.py
index 1e2463b3f..a2da3c178 100644
--- a/virtool/db/history.py
+++ b/virtool/db/history.py
@@ -116,7 +116,7 @@ async def add(db, method_name, old, new, description, user_id):
     return document
 
 
-async def find(db, req_query, base_query=None):
+async def find(db, req_query, base_query=None, unbuilt=False):
     data = await paginate(
         db.history,
         {},
Repository: virtool/virtool
Issue: Unable to edit sample rights

Attempting to change any of the sample rights results in a 
`PATCH http://localhost:9950/api/samples/:sampleId/rights 404 (Not Found)` error, with a `{id: "not_found", message: "Not found"}` response, even when that particular sampleId and group exists. 


Changed files:
virtool/api/samples.py

diff --git a/virtool/api/samples.py b/virtool/api/samples.py
index 321fc88fd..6d759e638 100644
--- a/virtool/api/samples.py
+++ b/virtool/api/samples.py
@@ -261,7 +261,7 @@ async def edit(req):
     return json_response(processed)
 
 
-@routes.post("/api/samples/rights", schema={
+@routes.patch("/api/samples/{sample_id}/rights", schema={
     "group": {"type": "string"},
     "all_read": {"type": "boolean"},
     "all_write": {"type": "boolean"},
Repository: virtool/virtool
Issue: Create Sample error

@igboyes 
After the latest fix, I seem to get the following error:
```
"POST /api/samples HTTP/1.1" 201 
Traceback (most recent call last):
  File "/home/naomis/Projects/virtool/virtool/jobs/job.py", line 75, in run method()
  File "/home/naomis/Projects/virtool/virtool/jobs/create_sample.py", line 96, in make_sample_dir analysis_path = os.path.join(self.sample_path, "analysis")
  File "/home/naomis/.virtualenvs/virtool/lib/python3.6/posixpath.py", line 80, in join a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run()
  File "/home/naomis/Projects/virtool/virtool/jobs/job.py", line 87, in run self.cleanup()
  File "/home/naomis/Projects/virtool/virtool/jobs/create_sample.py", line 318, in cleanup
    for file_id in self.files:
TypeError: 'NoneType' object is not iterable

```

Changed files:
virtool/jobs/create_sample.py

diff --git a/virtool/jobs/create_sample.py b/virtool/jobs/create_sample.py
index dd2038597..d2dbde41e 100644
--- a/virtool/jobs/create_sample.py
+++ b/virtool/jobs/create_sample.py
@@ -51,6 +51,7 @@ class CreateSample(virtool.jobs.job.Job):
 
         #: The ordered list of :ref:`stage methods <stage-methods>` that are called by the job.
         self._stage_list = [
+            self.check_db,
             self.make_sample_dir,
             self.trim_reads,
             self.save_trimmed,
Repository: virtool/virtool
Issue: bowtie_build stalling

Creating a new subtraction results in a job that seems to run forever at `progress: 0.67` without updates or 
 any errors.
Similarly with build index jobs; the cause seems to be the `bowtie_build` step.

Changed files:
virtool/jobs/job.py, virtool/api/uploads.py

diff --git a/virtool/api/uploads.py b/virtool/api/uploads.py
index 186733540..e605e2cae 100644
--- a/virtool/api/uploads.py
+++ b/virtool/api/uploads.py
@@ -7,7 +7,7 @@ import virtool.http.routes
 import virtool.utils
 from virtool.api.utils import invalid_query, json_response, not_found
 
-CHUNK_SIZE = 32768
+CHUNK_SIZE = 131072
 
 FILE_TYPES = [
     "reference",
diff --git a/virtool/jobs/job.py b/virtool/jobs/job.py
index 66d838a0f..0f168d3bc 100644
--- a/virtool/jobs/job.py
+++ b/virtool/jobs/job.py
@@ -109,8 +109,8 @@ class Job(multiprocessing.Process):
 
         self._process = subprocess.Popen(command, stdout=stdout, stderr=subprocess.PIPE, env=env)
 
-        stdout_thread = None
         stdout_queue = None
+        stdout_thread = None
 
         if stdout_handler:
             stdout_queue = queue.Queue()
@@ -142,11 +142,13 @@ class Job(multiprocessing.Process):
                 err = stderr_queue.get()
                 _stderr_handler(err)
 
-            alive = (stdout_thread and stdout_thread.is_alive()) or stderr_thread.is_alive()
+            # Continue to next iteration if queues are not empty (or stdout queue was not created).
+            if not stderr_queue.empty() or (stdout_queue and not stdout_queue.empty()):
+                continue
 
-            queue_empty = (stdout_queue and stdout_queue.empty()) and stderr_queue.empty()
+            alive = (stdout_thread and stdout_thread.is_alive()) or stderr_thread.is_alive()
 
-            if not alive and queue_empty and self._process.poll() is not None:
+            if not alive and self._process.poll() is not None:
                 break
 
         if self._process.returncode != 0:
Repository: virtool/virtool
Issue: Updating OTU schema issue

Attempting to add segments to the schema one-after-another results in replacing the existing segment with the added data. The length of the array never is greater than 1.

Refreshing the page between additions results and the segments being added properly and the schema growing in length.

Changed files:
virtool/db/otus.py

diff --git a/virtool/db/otus.py b/virtool/db/otus.py
index 302503ae7..b25437cf5 100644
--- a/virtool/db/otus.py
+++ b/virtool/db/otus.py
@@ -315,7 +315,7 @@ async def update_last_indexed_version(db, id_list, version):
 
 
 async def update_sequence_segments(db, old, new):
-    if old is None or new is None:
+    if old is None or new is None or "schema" not in old:
         return
 
     old_names = {s["name"] for s in old["schema"]}
Repository: virtool/virtool
Issue: Include reference in OTU projection

From @Miminko in #969:

> Websocket messages detailing OTU insert & update requires a reference field indicating which reference id the particular otu is associated with.

Changed files:
virtool/db/otus.py

diff --git a/virtool/db/otus.py b/virtool/db/otus.py
index 81d702222..0f8bd9d40 100644
--- a/virtool/db/otus.py
+++ b/virtool/db/otus.py
@@ -10,10 +10,11 @@ from virtool.api.utils import compose_regex_query, paginate
 
 PROJECTION = [
     "_id",
-    "name",
     "abbreviation",
-    "version",
-    "verified"
+    "name",
+    "reference",
+    "verified",
+    "version"
 ]
 
 SEQUENCE_PROJECTION = [