Repository: aboutcode-org/purldb (language: python)

Issue:
Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Reorganize visitor/mapper code
Currently we are putting code for the on-demand/priority queue in with the visitor code. It would be better if we separated the code. We should also decide which term we want to use for the on-demand/priority queue, since we use both currently. It is also worth considering if we want to merge the visitor and mapper code as well.
Reorganize visitor/mapper code
Currently we are putting code for the on-demand/priority queue in with the visitor code. It would be better if we separated the code. We should also decide which term we want to use for the on-demand/priority queue, since we use both currently. It is also worth considering if we want to merge the visitor and mapper code as well.
Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Scan queue updates
We are revamping the scan queue. The new process will have scancode.io workers get jobs from the purldb by visiting an API endpoint that returns the next available scan queue entry to be worked on. (`api/scan_queue/get_next_download_url`). When the worker is done with the scan, it will post the results back to `api/scan_queue/submit_scan_results`. 

TODO:

- [x] Guard indexing endpoints with special API key
- [x] Update scan queue to create new scan queue entries whenever we reindex a package, rather than reusing existing entry
- [x] Create scancode.io management command that gets jobs from `api/scan_queue/get_next_download_url` and sends results back
    - [x] Get download url
    - [x] Create project, get input, run pipeline
    - [x] Return results

Consider:

- [ ] POST scancode io worker info before getting download url so we can tell which worker took what job
Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14

Overhaul needed for Package scan requests and indexing
I've set up purldb and scancode.io locally, where I run `make run_visit` and `make run_map` to visit and map Maven packages. I then run `make request_scans` and `make process_scans` to get information on the Resources in the packages we visit and map. I've noticed that the scan requests we send off to scancode.io are for multiple versions of the same package. This causes a few problems:

- We cannot possibly scan every single package in the maven index to create our matching index
- In the case of the directory structure fingerprint, if there is little/no difference between the different versions of a package, then we are essentially scanning and indexing the same package repeatedly
- We can't populate our matching index with a single scancode.io instance, we need multiple instances so we can have multiple scans going at once

For the first two issues, we need to come up with a new way to group and index fingerprints. A starting idea would be to come up with a bit more general. Currently, we create directory fingerprints for every package we map. If two packages we index are the same package but different versions, then we may have the same fingerprints twice. We could do something along the lines of indexing fingerprints to a package in general, rather than to a specific package version.

For the second issue, we will have to flip the current scan queue request model. purldb will have a queue of packages that it wants scanned and it will be up to scancode.io to poll purldb to see what needs to be scanned. scancode.io would poll purldb, get the package that needs to be scanned, scan and fingerprint it, then send the results back to purldb. This issue is tracked at https://github.com/nexB/purldb/issues/14



Discussion / hints:
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!


There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!

There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!
There are my notes for an on-demand queue for requesting package information. This queue would take in a package URL, and the queue would figure out which handler works for the package url, and then look at the upstream repo for package information.

This is a queue that allows us to:

- Visit and map a package requested by the user
- Visit and map the other versions of the package requested by the user

How does (should) it work:

- Take a purl and create something similar to a ResourceURI
- Using the purl, look up in the right repo, then get pom info and map info as package
- Look up other versions of the package and map that package info

How should we start:

- Create ResourceURI-like model
- Create queue loop (we will add the ResourceURI in manually for now, in the future we will have an API or something that can take in requests)
- Queue loop only works on maven packages for now, in the future we will have that router thing like what we do with the visitors and mappers

https://repo1.maven.org/maven2/yom/yom/1.0-alpha-1/
`https://repo1.maven.org/maven2/<namespace>/<name>/<version>/<name>-<version>-<classifier>.jar`


We would first create a new Package in the packagedb with the initial information from maven, then we would download the sources jar of the package (if available), create a Package and index it. We would also have to do some sort of summarization (maybe use the license clarity score plugin?) on the scanned sources jar in order to get a copyright and license.
@JonoYang    all this looks great!  Question please: is the "ResourceURI" the same concept as the "Inferred URL" that we can see in scan results?  or does it include other cases?  (I guess I am asking for a somewhat precise definition of a ResourceURI.) 
@DennisClark 

ResourceURI is different than inferred URLs that are generated from purls. 

ResourceURI is a model that represents a Resource from the internet you can download. The value of these URIs are usually the download URLs for a Package. In special cases, URIs could point to an upstream repo's index, like https://repo1.maven.org/maven2/.index/nexus-maven-repository-index.gz or https://replicate.npmjs.com/registry/_changes?include_docs=true&limit=1000&since=0. These are called seed URIs. 

In purldb, we have visitors and mappers that work on these ResourceURIs. Visitors visit the seed URIs and create ResourceURIs for the packages it finds listed from the repo index. Mappers take the ResourceURIs created by visitors and creates new entries in the PackageDB for each ResourceURI.


@JonoYang   thanks, that all makes sense!

Fix patch:
diff --git a/Makefile b/Makefile
index 6ebc900c..e97c0294 100644
--- a/Makefile
+++ b/Makefile
@@ -122,12 +122,6 @@ run_visit: seed
 run_map:
 	${MANAGE} run_map
 
-request_scans:
-	${MANAGE} request_scans
-
-process_scans:
-	${MANAGE} process_scans
-
 test:
 	@echo "-> Run the test suite"
 	${ACTIVATE} DJANGO_SETTINGS_MODULE=purldb_project.settings ${PYTHON_EXE} -m pytest -vvs --ignore matchcode_pipeline --ignore matchcode_project --ignore purldb-toolkit --ignore packagedb/tests/test_throttling.py
diff --git a/docker-compose_purldb.yml b/docker-compose_purldb.yml
index bb1eb3be..261fa04e 100644
--- a/docker-compose_purldb.yml
+++ b/docker-compose_purldb.yml
@@ -81,32 +81,6 @@ services:
       - db
       - web # Ensure that potential db migrations run first
 
-  request_scan:
-    build: .
-    command: wait-for-it web:8000 -- python manage_purldb.py request_scans
-    env_file:
-      - docker_purldb.env
-    volumes:
-      - /etc/purldb/:/etc/purldb/
-    profiles:
-      - scan_queue
-    depends_on:
-      - db
-      - web
-
-  process_scan:
-    build: .
-    command: wait-for-it web:8000 -- python manage_purldb.py process_scans
-    env_file:
-      - docker_purldb.env
-    volumes:
-      - /etc/purldb/:/etc/purldb/
-    profiles:
-      - scan_queue
-    depends_on:
-      - db
-      - web
-
   priority_queue:
     build: .
     command: wait-for-it web:8000 -- python manage_purldb.py priority_queue
diff --git a/docker-compose_purldb_public.yml b/docker-compose_purldb_public.yml
index 4ffbf261..242243d8 100644
--- a/docker-compose_purldb_public.yml
+++ b/docker-compose_purldb_public.yml
@@ -74,32 +74,6 @@ services:
       - db
       - web # Ensure that potential db migrations run first
 
-  request_scan:
-    build: .
-    command: wait-for-it web:8000 -- python manage_purldb_public.py request_scans
-    env_file:
-      - docker_purldb.env
-    volumes:
-      - /etc/purldb/:/etc/purldb/
-    profiles:
-      - scan_queue
-    depends_on:
-      - db
-      - web
-
-  process_scan:
-    build: .
-    command: wait-for-it web:8000 -- python manage_purldb_public.py process_scans
-    env_file:
-      - docker_purldb.env
-    volumes:
-      - /etc/purldb/:/etc/purldb/
-    profiles:
-      - scan_queue
-    depends_on:
-      - db
-      - web
-
   priority_queue:
     build: .
     command: wait-for-it web:8000 -- python manage_purldb_public.py priority_queue
diff --git a/minecode/api.py b/minecode/api.py
index b282ef56..766f3e78 100644
--- a/minecode/api.py
+++ b/minecode/api.py
@@ -7,19 +7,24 @@
 # See https://aboutcode.org for more information about nexB OSS projects.
 #
 
-from rest_framework import serializers
-from rest_framework import status
-from rest_framework import viewsets
+import json
+
+from django.db import transaction
+from django.utils import timezone
+from packageurl import PackageURL
+from rest_framework import serializers, status, viewsets
 from rest_framework.decorators import action
+from rest_framework.permissions import IsAdminUser
 from rest_framework.response import Response
-from packageurl import PackageURL
 
 # UnusedImport here!
 # But importing the mappers and visitors module triggers routes registration
 from minecode import visitors  # NOQA
 from minecode import priority_router
-from minecode.models import ResourceURI
-from minecode.models import PriorityResourceURI
+from minecode.management.indexing import index_package
+from minecode.models import PriorityResourceURI, ResourceURI, ScannableURI
+from minecode.permissions import IsScanQueueWorkerAPIUser
+from minecode.utils import validate_uuid
 
 
 class ResourceURISerializer(serializers.ModelSerializer):
@@ -83,3 +88,147 @@ def index_package(self, request, *args, **kwargs):
             }
         # TODO: revisiting a package should be handled on another level, dependent on data we store
         return Response(message)
+
+
+class ScannableURISerializer(serializers.ModelSerializer):
+    class Meta:
+        model = ScannableURI
+        fields = '__all__'
+
+
+class ScannableURIViewSet(viewsets.ModelViewSet):
+    queryset = ScannableURI.objects.all()
+    serializer_class = ScannableURISerializer
+    permission_classes = [IsScanQueueWorkerAPIUser|IsAdminUser]
+
+    @action(detail=False, methods=['get'])
+    def get_next_download_url(self, request, *args, **kwargs):
+        """
+        Return download url for next Package on scan queue
+        """
+        with transaction.atomic():
+            scannable_uri = ScannableURI.objects.get_next_scannable()
+            if scannable_uri:
+                response = {
+                    'scannable_uri_uuid': scannable_uri.uuid,
+                    'download_url': scannable_uri.uri,
+                    'pipelines': scannable_uri.pipelines,
+                }
+                scannable_uri.scan_status = ScannableURI.SCAN_SUBMITTED
+                scannable_uri.scan_date = timezone.now()
+                scannable_uri.save()
+            else:
+                response = {
+                    'scannable_uri_uuid': '',
+                    'download_url': '',
+                    'pipelines': [],
+                }
+            return Response(response)
+
+    @action(detail=False, methods=['post'])
+    def update_status(self, request, *args, **kwargs):
+        """
+        Update the status of a ScannableURI with UUID of `scannable_uri_uuid`
+        with `scan_status`
+
+        If `scan_status` is 'failed', then a `scan_log` string is expected and
+        should contain the error messages for that scan.
+
+        If `scan_status` is 'scanned', then a `scan_results_file`,
+        `scan_summary_file`, and `project_extra_data` mapping are expected.
+        `scan_results_file`, `scan_summary_file`, and `project_extra_data` are
+        then used to update Package data and its Resources.
+        """
+        scannable_uri_uuid = request.data.get('scannable_uri_uuid')
+        scan_status = request.data.get('scan_status')
+        if not scannable_uri_uuid:
+            response = {
+                'error': 'missing scannable_uri_uuid'
+            }
+            return Response(response, status=status.HTTP_400_BAD_REQUEST)
+
+        if not scan_status:
+            response = {
+                'error': 'missing scan_status'
+            }
+            return Response(response, status=status.HTTP_400_BAD_REQUEST)
+
+        if not validate_uuid(scannable_uri_uuid):
+            response = {
+                'error': f'invalid scannable_uri_uuid: {scannable_uri_uuid}'
+            }
+            return Response(response, status=status.HTTP_400_BAD_REQUEST)
+
+        scannable_uri = ScannableURI.objects.get(uuid=scannable_uri_uuid)
+        scannable_uri_status = ScannableURI.SCAN_STATUSES_BY_CODE.get(scannable_uri.scan_status)
+        scan_status_code = ScannableURI.SCAN_STATUS_CODES_BY_SCAN_STATUS.get(scan_status)
+
+        if not scan_status_code:
+            msg = {
+                'error': f'invalid scan_status: {scan_status}'
+            }
+            return Response(msg, status=status.HTTP_400_BAD_REQUEST)
+
+        if scannable_uri.scan_status in [
+            ScannableURI.SCAN_INDEXED,
+            ScannableURI.SCAN_FAILED,
+            ScannableURI.SCAN_TIMEOUT,
+            ScannableURI.SCAN_INDEX_FAILED,
+        ]:
+            response = {
+                'error': f'cannot update status for scannable_uri {scannable_uri_uuid}: '
+                         f'scannable_uri has finished with status "{scannable_uri_status}"'
+            }
+            return Response(response, status=status.HTTP_400_BAD_REQUEST)
+
+        if scan_status == scannable_uri_status:
+            response = {
+                'error': f'cannot update status for scannable_uri {scannable_uri_uuid}: '
+                         f'scannable_uri status is already "{scannable_uri_status}"'
+            }
+            return Response(response, status=status.HTTP_400_BAD_REQUEST)
+
+        if scan_status == 'failed':
+            scan_log = request.data.get('scan_log')
+            scannable_uri.scan_error = scan_log
+            scannable_uri.scan_status = ScannableURI.SCAN_FAILED
+            scannable_uri.wip_date = None
+            scannable_uri.save()
+            msg = {
+                'status': f'updated scannable_uri {scannable_uri_uuid} scan_status to {scan_status}'
+            }
+
+        elif scan_status == 'scanned':
+            scan_results_file = request.data.get('scan_results_file')
+            scan_summary_file = request.data.get('scan_summary_file')
+            project_extra_data = request.data.get('project_extra_data')
+            scan_data = json.load(scan_results_file)
+            summary_data = json.load(scan_summary_file)
+            project_extra_data = json.loads(project_extra_data)
+
+            scannable_uri.scan_status = ScannableURI.SCAN_COMPLETED
+
+            indexing_errors = index_package(
+                scannable_uri,
+                scannable_uri.package,
+                scan_data,
+                summary_data,
+                project_extra_data,
+                reindex=scannable_uri.reindex_uri,
+            )
+            if indexing_errors:
+                scannable_uri.scan_status = ScannableURI.SCAN_INDEX_FAILED
+                scannable_uri.index_error = indexing_errors
+                msg = {
+                    'error': f'scan index failed for scannable_uri {scannable_uri_uuid}'
+                }
+                return Response(msg, status=status.HTTP_400_BAD_REQUEST)
+            else:
+                scannable_uri.scan_status = ScannableURI.SCAN_INDEXED
+                msg = {
+                    'status': f'scan indexed for scannable_uri {scannable_uri_uuid}'
+                }
+            scannable_uri.wip_date = None
+            scannable_uri.save()
+
+        return Response(msg)
diff --git a/minecode/management/commands/create-scan-queue-worker-user.py b/minecode/management/commands/create-scan-queue-worker-user.py
new file mode 100644
index 00000000..76e7b928
--- /dev/null
+++ b/minecode/management/commands/create-scan-queue-worker-user.py
@@ -0,0 +1,30 @@
+#
+# Copyright (c) nexB Inc. and others. All rights reserved.
+# purldb is a trademark of nexB Inc.
+# SPDX-License-Identifier: Apache-2.0
+# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
+# See https://github.com/nexB/purldb for support or download.
+# See https://aboutcode.org for more information about nexB OSS projects.
+#
+
+from django.contrib.auth.models import Group
+from minecode.management.user_creation import CreateUserCommand
+
+
+class Command(CreateUserCommand):
+    help = 'Create a user and generate an API key for a scan queue worker'
+
+    def handle(self, *args, **options):
+        username = options['username']
+        interactive = options['interactive']
+        verbosity = options['verbosity']
+        user = self.create_user(
+            username=username,
+            interactive=interactive,
+            verbosity=verbosity
+        )
+        # Add user to `scan_queue_workers` group
+        scan_queue_workers_group, _ = Group.objects.get_or_create(name='scan_queue_workers')
+        scan_queue_workers_group.user_set.add(user)
+        msg = f'User {username} added to `scan_queue_workers` group'
+        self.stdout.write(msg, self.style.SUCCESS)
diff --git a/minecode/management/commands/create-user.py b/minecode/management/commands/create-user.py
new file mode 100644
index 00000000..64b8b238
--- /dev/null
+++ b/minecode/management/commands/create-user.py
@@ -0,0 +1,24 @@
+#
+# Copyright (c) nexB Inc. and others. All rights reserved.
+# purldb is a trademark of nexB Inc.
+# SPDX-License-Identifier: Apache-2.0
+# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
+# See https://github.com/nexB/purldb for support or download.
+# See https://aboutcode.org for more information about nexB OSS projects.
+#
+
+from minecode.management.user_creation import CreateUserCommand
+
+
+class Command(CreateUserCommand):
+    help = 'Create a user and generate an API key for a scan queue worker'
+
+    def handle(self, *args, **options):
+        username = options['username']
+        interactive = options['interactive']
+        verbosity = options['verbosity']
+        self.create_user(
+            username=username,
+            interactive=interactive,
+            verbosity=verbosity
+        )
diff --git a/minecode/management/commands/import_queue.py b/minecode/management/commands/import_queue.py
index 55862921..9003de5f 100644
--- a/minecode/management/commands/import_queue.py
+++ b/minecode/management/commands/import_queue.py
@@ -29,7 +29,6 @@
 from minecode.visitors.maven import get_artifact_sha1
 from minecode.model_utils import merge_or_create_package
 from packagedcode.models import PackageData
-from packagedb.models import Package
 from minecode.visitors.maven import determine_namespace_name_version_from_url
 
 
diff --git a/minecode/management/commands/manage_scans.py b/minecode/management/commands/manage_scans.py
new file mode 100644
index 00000000..a361f227
--- /dev/null
+++ b/minecode/management/commands/manage_scans.py
@@ -0,0 +1,169 @@
+#
+# Copyright (c) 2018 by nexB, Inc. http://www.nexb.com/ - All rights reserved.
+#
+
+import time
+import logging
+import signal
+import sys
+
+from django.db import transaction
+from django.utils import timezone
+
+
+from minecode.models import ScannableURI
+
+from minecode.management.commands import VerboseCommand
+
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(stream=sys.stdout)
+logger.setLevel(logging.INFO)
+
+
+# sleep duration in seconds when the queue is empty
+SLEEP_WHEN_EMPTY = 1
+
+
+class ScanningCommand(VerboseCommand):
+    """
+    Base command class for processing ScannableURIs.
+    """
+    # subclasses must override
+    logger = None
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            '--exit-on-empty',
+            dest='exit_on_empty',
+            default=False,
+            action='store_true',
+            help='Do not loop forever. Exit when the queue is empty.')
+
+        parser.add_argument(
+            '--max-uris',
+            dest='max_uris',
+            default=0,
+            action='store',
+            help='Limit the number of Scannable URIs processed to a maximum number. '
+                 '0 means no limit. Used only for testing.')
+
+    def handle(self, *args, **options):
+        exit_on_empty = options.get('exit_on_empty')
+        max_uris = options.get('max_uris', 0)
+
+        uris_counter = self.process_scans(
+            exit_on_empty=exit_on_empty,
+            max_uris=max_uris,
+            # Pass options to allow subclasses to add their own options
+            options=options
+        )
+        self.stdout.write('Processed {} ScannableURI.'.format(uris_counter))
+
+    @classmethod
+    def process_scans(cls, exit_on_empty=False, max_uris=0, **kwargs):
+        """
+        Run an infinite scan processing loop. Return a processed URis count.
+
+        Get the next available candidate ScannableURI and request a scan from
+        ScanCode.io. Loops forever and sleeps a short while if there are no
+        ScannableURI left to scan.
+        """
+        uris_counter = 0
+        sleeping = False
+
+        while True:
+            # Wait before processing anything
+            time.sleep(10)
+
+            if cls.MUST_STOP:
+                cls.logger.info('Graceful exit of the scan processing loop.')
+                break
+
+            if max_uris and uris_counter >= max_uris:
+                cls.logger.info('max_uris requested reached: exiting scan processing loop.')
+                break
+
+            scannable_uri = cls.get_next_uri()
+
+            if not scannable_uri:
+                if exit_on_empty:
+                    cls.logger.info('exit-on-empty requested: No more scannable URIs, exiting...')
+                    break
+
+                # Only log a single message when we go to sleep
+                if not sleeping:
+                    sleeping = True
+                    cls.logger.info('No more scannable URIs, sleeping for at least {} seconds...'.format(SLEEP_WHEN_EMPTY))
+
+                time.sleep(SLEEP_WHEN_EMPTY)
+                continue
+
+            cls.logger.info('Processing scannable URI: {}'.format(scannable_uri))
+
+            cls.process_scan(scannable_uri, **kwargs)
+            uris_counter += 1
+            sleeping = False
+
+        return uris_counter
+
+    @classmethod
+    def get_next_uri(self):
+        """
+        Return a locked ScannableURI for processing.
+        Subclasses must implement
+
+        Typically something like:
+            with transaction.atomic():
+                scannable_uri = ScannableURI.objects.get_next_scannable()
+        """
+        pass
+
+    @classmethod
+    def process_scan(scannable_uri, **kwargs):
+        """
+        Process a single `scannable_uri` ScannableURI. Subclasses must implement.
+        If sucessfully processed the ScannableURI must be updated accordingly.
+        """
+        pass
+
+
+class Command(ScanningCommand):
+
+    logger = logger
+
+    help = ('Check scancode.io requested scans for status then fetch and process '
+            'completed scans for indexing and updates.')
+
+    def handle(self, *args, **options):
+        logger.setLevel(self.get_verbosity(**options))
+        ScanningCommand.handle(self, *args, **options)
+
+    @classmethod
+    def get_next_uri(self):
+        with transaction.atomic():
+            scannable_uri = ScannableURI.objects.get_next_processable()
+        return scannable_uri
+
+    @classmethod
+    def process_scan(cls, scannable_uri, get_scan_info_save_loc='', get_scan_data_save_loc='', **kwargs):
+        """
+        Manage a ScannableURI based on its status.
+        - For submitted but not completed scans, check the timestamp of when the scan was submitted, if it has been past some time, then we set the scan as timed out
+        - For timed out scans, we set that as failed and then create a new one?
+        """
+        logger.info('Checking scan for URI: {}'.format(scannable_uri))
+
+        if scannable_uri.scan_status in (ScannableURI.SCAN_SUBMITTED, ScannableURI.SCAN_IN_PROGRESS):
+            scan_duration = timezone.now() - scannable_uri.scan_date
+            scan_duration_hours = scan_duration.seconds / (60 * 60)
+
+            if scan_duration_hours > 2:
+                scannable_uri.scan_status = ScannableURI.SCAN_TIMEOUT
+                scannable_uri.wip_date = None
+                scannable_uri.save()
+                logger.info('Scan for URI has timed out: {}'.format(scannable_uri))
+
+
+# support graceful death when used as a service
+signal.signal(signal.SIGTERM, Command.stop_handler)
diff --git a/minecode/management/commands/maven_crawler.py b/minecode/management/commands/maven_crawler.py
index 30c8f360..51c2be96 100644
--- a/minecode/management/commands/maven_crawler.py
+++ b/minecode/management/commands/maven_crawler.py
@@ -10,8 +10,8 @@
 import logging
 import sys
 
-from minecode.management.commands import VerboseCommand
 from minecode.visitors.maven import crawl_maven_repo_from_root
+from minecode.management.commands import VerboseCommand
 
 
 logger = logging.getLogger(__name__)
diff --git a/minecode/management/commands/priority_queue.py b/minecode/management/commands/priority_queue.py
index 04dda7f2..ca7d9382 100644
--- a/minecode/management/commands/priority_queue.py
+++ b/minecode/management/commands/priority_queue.py
@@ -105,19 +105,6 @@ def handle(self, *args, **options):
         return processed_counter
 
 
-def add_package_to_scan_queue(package):
-    """
-    Add a Package `package` to the scan queue
-    """
-    uri = package.download_url
-    _, scannable_uri_created = ScannableURI.objects.get_or_create(
-        uri=uri,
-        package=package,
-    )
-    if scannable_uri_created:
-        logger.debug(' + Inserted ScannableURI\t: {}'.format(uri))
-
-
 def process_request(priority_resource_uri, _priority_router=priority_router):
     purl_to_visit = priority_resource_uri.uri
     source_purl = priority_resource_uri.source_uri
diff --git a/minecode/management/commands/process_scans.py b/minecode/management/commands/process_scans.py
deleted file mode 100644
index f8566c4c..00000000
--- a/minecode/management/commands/process_scans.py
+++ /dev/null
@@ -1,282 +0,0 @@
-#
-# Copyright (c) 2018 by nexB, Inc. http://www.nexb.com/ - All rights reserved.
-#
-
-import logging
-import signal
-import sys
-import traceback
-
-from django.db import transaction
-
-from licensedcode.cache import build_spdx_license_expression
-from packagedcode.utils import combine_expressions
-
-from matchcode.models import ApproximateDirectoryContentIndex
-from matchcode.models import ApproximateDirectoryStructureIndex
-from matchcode.models import ExactFileIndex
-from minecode.management import scanning
-from minecode.management.commands import get_error_message
-from minecode.models import ScannableURI
-from minecode.model_utils import merge_or_create_resource
-
-
-logger = logging.getLogger(__name__)
-logging.basicConfig(stream=sys.stdout)
-logger.setLevel(logging.INFO)
-
-
-class Command(scanning.ScanningCommand):
-
-    logger = logger
-
-    help = ('Check scancode.io requested scans for status then fetch and process '
-            'completed scans for indexing and updates.')
-
-    def handle(self, *args, **options):
-        logger.setLevel(self.get_verbosity(**options))
-        scanning.ScanningCommand.handle(self, *args, **options)
-
-    @classmethod
-    def get_next_uri(self):
-        with transaction.atomic():
-            scannable_uri = ScannableURI.objects.get_next_processable()
-        return scannable_uri
-
-    @classmethod
-    def process_scan(cls, scannable_uri, get_scan_info_save_loc='', get_scan_data_save_loc='', **kwargs):
-        """
-        Process a ScannableURI based on its status.
-        - For requested but not completed scans, check remote status and
-          update status and timestamps accordingly.
-        - For completed scans, fetch the scan, then procpythess the scan results
-          to update the PackageDB as needed. Update status and timestamps accordingly
-        """
-        logger.info('Checking or processing scan for URI: {}'.format(scannable_uri))
-
-        scan_info = scanning.get_scan_info(
-            scannable_uri.scan_uuid,
-            api_url=cls.api_url,
-            api_auth_headers=cls.api_auth_headers,
-            get_scan_info_save_loc=get_scan_info_save_loc
-        )
-        rescan = scannable_uri.rescan_uri
-
-        if scannable_uri.scan_status in (ScannableURI.SCAN_SUBMITTED, ScannableURI.SCAN_IN_PROGRESS):
-            scannable_uri.scan_status = get_scan_status(scan_info)
-        elif scannable_uri.scan_status in (ScannableURI.SCAN_COMPLETED,):
-            scan_index_errors = []
-            try:
-                logger.info('Processing scan for URI: {}'.format(scannable_uri))
-
-                package = scannable_uri.package
-                input_size = scan_info.size
-                if input_size:
-                    computed_timeout = ((input_size / 1000000) / 2) * 60
-                    timeout = max(computed_timeout, scanning.REQUEST_TIMEOUT)
-                else:
-                    timeout = scanning.REQUEST_TIMEOUT
-                scan_data = scanning.get_scan_data(
-                    scannable_uri.scan_uuid,
-                    api_url=cls.api_url,
-                    api_auth_headers=cls.api_auth_headers,
-                    timeout=timeout,
-                    get_scan_data_save_loc=get_scan_data_save_loc
-                )
-                indexing_errors = index_package_files(package, scan_data, reindex=rescan)
-                scan_index_errors.extend(indexing_errors)
-
-                summary = scanning.get_scan_summary(
-                    scannable_uri.scan_uuid,
-                    api_url=cls.api_url,
-                    api_auth_headers=cls.api_auth_headers,
-                    get_scan_data_save_loc=get_scan_data_save_loc
-                )
-
-                declared_license_expression = summary.get('declared_license_expression')
-                declared_license_expression_spdx = None
-                if declared_license_expression:
-                    declared_license_expression_spdx = build_spdx_license_expression(declared_license_expression)
-
-                other_license_expressions = summary.get('other_license_expressions', [])
-                other_license_expressions = [l['value'] for l in other_license_expressions if l['value']]
-                other_license_expression = combine_expressions(other_license_expressions)
-
-                copyright = ''
-                declared_holder = summary.get('declared_holder')
-                if declared_holder:
-                    copyright = f'Copyright (c) {declared_holder}'
-
-                values_by_updateable_fields = {
-                    'sha1': scan_info.sha1,
-                    'sha256': scan_info.sha256,
-                    'sha512': scan_info.sha512,
-                    'summary': summary,
-                    'declared_license_expression': declared_license_expression,
-                    'declared_license_expression_spdx': declared_license_expression_spdx,
-                    'other_license_expression': other_license_expression,
-                    'copyright': copyright,
-                }
-
-                updated_fields = []
-                for field, value in values_by_updateable_fields.items():
-                    p_val = getattr(package, field)
-                    if (
-                        (not p_val and value)
-                        or rescan
-                    ):
-                        setattr(package, field, value)
-                        entry = dict(
-                            field=field,
-                            old_value=p_val,
-                            new_value=value,
-                        )
-                        updated_fields.append(entry)
-
-                if updated_fields:
-                    data = {
-                        'updated_fields': updated_fields,
-                    }
-                    package.append_to_history(
-                        'Package field values have been updated.',
-                        data=data,
-                        save=True,
-                    )
-
-                scannable_uri.scan_status = ScannableURI.SCAN_INDEXED
-                if rescan:
-                    scannable_uri.rescan = False
-                    scannable_uri.priority = 0
-
-            except Exception as e:
-                traceback_message = traceback.format_exc()
-                error_message = traceback_message + '\n'
-                # TODO: We should rerun the specific indexers that have failed
-                if scan_index_errors:
-                    error_message += '\n'.join(scan_index_errors)
-                scannable_uri.index_error = error_message
-                scannable_uri.scan_status = ScannableURI.SCAN_INDEX_FAILED
-
-        scannable_uri.wip_date = None
-        scannable_uri.save()
-
-
-# support graceful death when used as a service
-signal.signal(signal.SIGTERM, Command.stop_handler)
-
-
-def get_scan_status(scan_object):
-    """
-    Return a ScannableURI status from scan_object Scan
-    """
-    if scan_object.not_started or scan_object.queued:
-        scan_status = ScannableURI.SCAN_SUBMITTED
-    elif scan_object.running:
-        scan_status = ScannableURI.SCAN_IN_PROGRESS
-    elif scan_object.failure or scan_object.stopped or scan_object.stale:
-        scan_status = ScannableURI.SCAN_FAILED
-    elif scan_object.success:
-        scan_status = ScannableURI.SCAN_COMPLETED
-    else:
-        # TODO: Consider not raising an exception
-        raise Exception('Unknown scancode.io status')
-    return scan_status
-
-
-def update_package_checksums(package, scan_object):
-    """
-    Create a new Resource entry for `package` Package if its checksums have been updated
-
-    Return a list of scan error messages
-    """
-    scan_index_errors = []
-    try:
-        updated = _update_package_checksums(package, scan_object)
-    except Exception as e:
-        msg = get_error_message(e)
-        scan_index_errors.append(msg)
-        logger.error(msg)
-    return scan_index_errors
-
-
-def _update_package_checksums(package, scan_object):
-    """
-    Update and save `package` Package checksums with data from `scan_object` Scan.
-
-    Return True if the package was updated.
-    """
-    updated = False
-    if ((package.sha1 and package.sha1 != scan_object.sha1) or
-            (package.md5 and package.md5 != scan_object.md5) or
-            (package.size and package.size != scan_object.size)):
-        raise Exception(
-            'Inconsistent checksum or size collected from scan uuid: {} for Package {}'
-            .format(scan_object.uuid, package.uuid)
-        )
-
-    if not package.sha1:
-        package.sha1 = scan_object.sha1
-        updated = True
-    if not package.md5:
-        package.md5 = scan_object.md5
-        updated = True
-    if not package.size:
-        package.size = scan_object.size
-        updated = True
-    if updated:
-        package.save()
-    return updated
-
-
-def index_package_files(package, scan_data, reindex=False):
-    """
-    Index scan data for `package` Package.
-
-    Return a list of scan index errors messages
-
-    If `reindex` is True, then all fingerprints related to `package` will be
-    deleted and recreated from `scan_data`.
-    """
-    if reindex:
-        logger.info(f'Deleting fingerprints and Resources related to {package.package_url}')
-        package.approximatedirectorycontentindex_set.all().delete()
-        package.approximatedirectorystructureindex_set.all().delete()
-        package.exactfileindex_set.all().delete()
-        package.resources.all().delete()
-
-    scan_index_errors = []
-    try:
-        logger.info(f'Indexing Resources and fingerprints related to {package.package_url} from scan data')
-        for resource in scan_data.get('files', []):
-            r, _, _ = merge_or_create_resource(package, resource)
-            path = r.path
-            sha1 = r.sha1
-            if sha1:
-                _, _ = ExactFileIndex.index(
-                    sha1=sha1,
-                    package=package
-                )
-
-            resource_extra_data = resource.get('extra_data', {})
-            directory_content_fingerprint = resource_extra_data.get('directory_content', '')
-            directory_structure_fingerprint = resource_extra_data.get('directory_structure', '')
-
-            if directory_content_fingerprint:
-                _, _ = ApproximateDirectoryContentIndex.index(
-                    directory_fingerprint=directory_content_fingerprint,
-                    resource_path=path,
-                    package=package,
-                )
-            if directory_structure_fingerprint:
-                _, _ = ApproximateDirectoryStructureIndex.index(
-                    directory_fingerprint=directory_structure_fingerprint,
-                    resource_path=path,
-                    package=package,
-                )
-
-    except Exception as e:
-        msg = get_error_message(e)
-        scan_index_errors.append(msg)
-        logger.error(msg)
-
-    return scan_index_errors
diff --git a/minecode/management/commands/request_scans.py b/minecode/management/commands/request_scans.py
deleted file mode 100644
index f781b1b9..00000000
--- a/minecode/management/commands/request_scans.py
+++ /dev/null
@@ -1,104 +0,0 @@
-#
-# Copyright (c) 2018 by nexB, Inc. http://www.nexb.com/ - All rights reserved.
-#
-
-from __future__ import absolute_import
-from __future__ import unicode_literals
-
-import logging
-import signal
-import sys
-
-from django.db import transaction
-from django.db.models import Q
-from django.utils import timezone
-
-from minecode.management import scanning
-from minecode.management.commands import get_error_message
-from minecode.models import ScannableURI
-
-
-logger = logging.getLogger(__name__)
-logging.basicConfig(stream=sys.stdout)
-logger.setLevel(logging.INFO)
-
-
-class Command(scanning.ScanningCommand):
-    logger = logger
-
-    help = 'Request scans for ScannableURIs from scancode.io.'
-
-    def add_arguments(self, parser):
-        parser.add_argument(
-            '--max-scan-requests',
-            dest='max_scan_requests',
-            action='store',
-            help='Limit the number of scan requests that can be made',
-            default=3,
-        )
-
-    def handle(self, *args, **options):
-        self.logger.setLevel(self.get_verbosity(**options))
-        scanning.ScanningCommand.handle(self, *args, **options)
-
-    @classmethod
-    def get_next_uri(self):
-        with transaction.atomic():
-            scannable_uri = ScannableURI.objects.get_next_scannable()
-        return scannable_uri
-
-    @classmethod
-    def process_scan(cls, scannable_uri, options, response_save_loc='', **kwargs):
-        """
-        Request a ScanCode.io scan for a `scannable_uri` ScannableURI.
-        """
-        uri = scannable_uri.uri
-        max_scan_requests = options.get('max_scan_requests', 3)
-        if isinstance(max_scan_requests, int):
-            submitted_and_in_progress = ScannableURI.objects.filter(
-                Q(scan_status=ScannableURI.SCAN_SUBMITTED) | Q(scan_status=ScannableURI.SCAN_IN_PROGRESS)
-            ).count()
-
-            if submitted_and_in_progress >= max_scan_requests:
-                cls.logger.info(f'Max scan requests reached: {max_scan_requests} Skipping URI "{uri}"')
-                return
-
-        scan_errors = []
-        scancodeio_uuid = scan_error = None
-
-        try:
-            cls.logger.info('Requesting scan from ScanCode.io for URI: "{uri}"'.format(**locals()))
-            scan = scanning.submit_scan(
-                uri,
-                scannable_uri.package,
-                api_url=cls.api_url,
-                api_auth_headers=cls.api_auth_headers,
-                response_save_loc=response_save_loc
-            )
-            scancodeio_uuid = scan.uuid
-
-        except Exception as e:
-            msg = 'Scan request error for URI: "{uri}"'.format(**locals())
-            msg += '\n'.format(scannable_uri.uri)
-            msg += get_error_message(e)
-            scan_errors.append(msg)
-            cls.logger.error(msg)
-
-        finally:
-            # Flag the processed scannable_uri as completed
-            scannable_uri.scan_status = ScannableURI.SCAN_SUBMITTED
-            scannable_uri.scan_request_date = timezone.now()
-            scannable_uri.scan_uuid = scancodeio_uuid
-            scannable_uri.wip_date = None
-
-            if scan_errors:
-                cls.logger.debug(' ! Scan request errors.')
-                scannable_uri.scan_error = '\n'.join(scan_errors)[:5000]
-            else:
-                cls.logger.debug(' + Scan requested OK.')
-
-            scannable_uri.save()
-
-
-# support graceful death when used as a service
-signal.signal(signal.SIGTERM, Command.stop_handler)
diff --git a/minecode/management/indexing.py b/minecode/management/indexing.py
new file mode 100644
index 00000000..593e53d7
--- /dev/null
+++ b/minecode/management/indexing.py
@@ -0,0 +1,118 @@
+#
+# Copyright (c) 2018 by nexB, Inc. http://www.nexb.com/ - All rights reserved.
+#
+from matchcode.models import ApproximateDirectoryContentIndex
+from matchcode.models import ApproximateDirectoryStructureIndex
+from matchcode.models import ExactFileIndex
+from minecode.management.commands import get_error_message
+import logging
+import sys
+from minecode.model_utils import merge_or_create_resource
+from packagedcode.utils import combine_expressions
+import traceback
+from minecode.models import ScannableURI
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(stream=sys.stdout)
+logger.setLevel(logging.INFO)
+
+
+def index_package_files(package, scan_data, reindex=False):
+    """
+    Index scan data for `package` Package.
+
+    Return a list of scan index errors messages
+
+    If `reindex` is True, then all fingerprints related to `package` will be
+    deleted and recreated from `scan_data`.
+    """
+    if reindex:
+        logger.info(f'Deleting fingerprints and Resources related to {package.package_url}')
+        package.approximatedirectorycontentindex_set.all().delete()
+        package.approximatedirectorystructureindex_set.all().delete()
+        package.exactfileindex_set.all().delete()
+        package.resources.all().delete()
+
+    scan_index_errors = []
+    try:
+        logger.info(f'Indexing Resources and fingerprints related to {package.package_url} from scan data')
+        for resource in scan_data.get('files', []):
+            r, _, _ = merge_or_create_resource(package, resource)
+            path = r.path
+            sha1 = r.sha1
+            if sha1:
+                _, _ = ExactFileIndex.index(
+                    sha1=sha1,
+                    package=package
+                )
+
+            resource_extra_data = resource.get('extra_data', {})
+            directory_content_fingerprint = resource_extra_data.get('directory_content', '')
+            directory_structure_fingerprint = resource_extra_data.get('directory_structure', '')
+
+            if directory_content_fingerprint:
+                _, _ = ApproximateDirectoryContentIndex.index(
+                    directory_fingerprint=directory_content_fingerprint,
+                    resource_path=path,
+                    package=package,
+                )
+            if directory_structure_fingerprint:
+                _, _ = ApproximateDirectoryStructureIndex.index(
+                    directory_fingerprint=directory_structure_fingerprint,
+                    resource_path=path,
+                    package=package,
+                )
+
+    except Exception as e:
+        msg = get_error_message(e)
+        scan_index_errors.append(msg)
+        logger.error(msg)
+
+    return scan_index_errors
+
+
+def index_package(scannable_uri, package, scan_data, summary_data, project_extra_data, reindex=False):
+    scan_index_errors = []
+    try:
+        indexing_errors = index_package_files(package, scan_data, reindex=reindex)
+        scan_index_errors.extend(indexing_errors)
+        declared_license_expression = summary_data.get('declared_license_expression')
+        other_license_expressions = summary_data.get('other_license_expressions', [])
+        other_license_expressions = [l['value'] for l in other_license_expressions if l['value']]
+        other_license_expression = combine_expressions(other_license_expressions)
+
+        copyright = ''
+        declared_holder = summary_data.get('declared_holder')
+        if declared_holder:
+            copyright = f'Copyright (c) {declared_holder}'
+
+        checksums_and_size_by_field = {
+            k: v
+            for k, v in project_extra_data.items()
+            if k in [
+                'md5','sha1', 'size', 'sha256', 'sha512', 'filename'
+            ]
+        }
+        values_by_updateable_fields = {
+            'summary': summary_data,
+            'declared_license_expression': declared_license_expression,
+            'other_license_expression': other_license_expression,
+            'copyright': copyright,
+            **checksums_and_size_by_field
+        }
+        _, updated_fields = package.update_fields(save=True, **values_by_updateable_fields)
+        updated_fields = ', '.join(updated_fields)
+        message = f'Updated fields for Package {package.purl}: {updated_fields}'
+        logger.info(message)
+        scannable_uri.scan_status = ScannableURI.SCAN_INDEXED
+        scannable_uri.save()
+    except Exception as e:
+        traceback_message = traceback.format_exc()
+        error_message = traceback_message + '\n'
+        # TODO: We should rerun the specific indexers that have failed
+        if scan_index_errors:
+            error_message += '\n'.join(scan_index_errors)
+        logger.error(error_message)
+        scannable_uri.index_error = error_message
+        scannable_uri.scan_status = ScannableURI.SCAN_INDEX_FAILED
+        scannable_uri.save()
diff --git a/minecode/management/scanning.py b/minecode/management/scanning.py
deleted file mode 100644
index 9639649e..00000000
--- a/minecode/management/scanning.py
+++ /dev/null
@@ -1,482 +0,0 @@
-#
-# Copyright (c) 2018 by nexB, Inc. http://www.nexb.com/ - All rights reserved.
-#
-from uuid import uuid4
-import hashlib
-import logging
-import sys
-import time
-import json
-
-import attr
-import requests
-
-from django.conf import settings
-
-from minecode.management.commands import VerboseCommand
-
-logger = logging.getLogger(__name__)
-logging.basicConfig(stream=sys.stdout)
-logger.setLevel(logging.INFO)
-
-# sleep duration in seconds when the queue is empty
-SLEEP_WHEN_EMPTY = 1
-
-# in seconds
-REQUEST_TIMEOUT = 120
-
-# Only SCANCODEIO_URL can be provided through setting
-SCANCODEIO_URL = settings.SCANCODEIO_URL
-SCANCODEIO_API_URL = f'{SCANCODEIO_URL.rstrip("/")}/api/' if SCANCODEIO_URL else None
-SCANCODEIO_API_URL_PROJECTS = f'{SCANCODEIO_API_URL}projects/' if SCANCODEIO_API_URL else None
-
-# Authentication with single API key
-SCANCODEIO_API_KEY = settings.SCANCODEIO_API_KEY
-SCANCODEIO_AUTH_HEADERS = {
-    'Authorization': f'Token {SCANCODEIO_API_KEY}'
-} if SCANCODEIO_API_KEY else {}
-
-
-@attr.attrs(slots=True)
-class Scan(object):
-    """
-    Represent a scan record as returned by the ScanCode.io API /scans/<UUID>
-    endpoint.
-    """
-    # this is the API endpoint full URL:
-    # "url": "https://scancode.io/api/scans/ac85c2f0-09b9-4ca1-b0e4-91523a636ccf/",
-    url = attr.ib(default=None)
-    # this is the UUDI for for scan:
-    # "uuid": "ac85c2f0-09b9-4ca1-b0e4-91523a636ccf",
-    uuid = attr.ib(default=None)
-    # The UUID for the scan run
-    run_uuid = attr.ib(default=None)
-    # the actual URI being scanned:
-    # "uri": "https://repo1.maven.org/maven2/io/github/subiyacryolite/jds/3.0.1/jds-3.0.1-sources.jar",
-    uri = attr.ib(default=None)
-    # set at creation of a scan request
-    # "created_date": "2018-06-19T08:33:34.953429Z",
-    created_date = attr.ib(default=None)
-    # set at start of the actual fetch+scan:
-    # "task_start_date": null,
-    task_start_date = attr.ib(default=None)
-    # set at end of scanning:
-    # "task_end_date": null,
-    task_end_date = attr.ib(default=None)
-    # null and then 0 on success or 1 or else on failure
-    task_exitcode = attr.ib(default=None)
-    # ignore for now
-    # task_output=attr.ib(default=None)
-    # "status": 'not started yet', 'failed', 'in progress', 'completed'
-    status = attr.ib(default=None)
-    # as a time stamp
-    execution_time = attr.ib(default=None)
-    md5 = attr.ib(default=None)
-    sha1 = attr.ib(default=None)
-    sha256 = attr.ib(default=None)
-    sha512 = attr.ib(default=None)
-    sha1_git = attr.ib(default=None)
-    filename = attr.ib(default=None)
-    size = attr.ib(default=None)
-
-    @classmethod
-    def from_response(cls, url, uuid, runs, input_sources, extra_data={}, **kwargs):
-        """
-        Return a Scan object built from an API response data arguments.
-        """
-        run_data = {}
-        if len(runs) > 0:
-            run_data = runs[0]
-
-        run_uuid = run_data.get("uuid")
-        created_date = run_data.get("created_date")
-        task_start_date = run_data.get("task_start_date")
-        task_end_date = run_data.get("task_end_date")
-        task_exitcode = run_data.get("task_exitcode")
-        status = run_data.get("status")
-        execution_time = run_data.get('execution_time')
-
-        if len(input_sources) > 0:
-            uri = input_sources[0]["download_url"]
-
-        md5 = extra_data.get('md5')
-        sha1 = extra_data.get('sha1')
-        sha256 = extra_data.get('sha256')
-        sha512 = extra_data.get('sha512')
-        sha1_git = extra_data.get('sha1_git')
-        filename = extra_data.get('filename')
-        size = extra_data.get('size')
-
-        return Scan(
-            url=url, uuid=uuid, run_uuid=run_uuid, uri=uri,
-            created_date=created_date, task_start_date=task_start_date,
-            task_end_date=task_end_date, task_exitcode=task_exitcode,
-            status=status, execution_time=execution_time,
-            md5=md5, sha1=sha1, sha256=sha256, sha512=sha512,
-            sha1_git=sha1_git, filename=filename, size=size
-        )
-
-    @property
-    def results_url(self):
-        url = self.url.rstrip('/')
-        return f'{url}/results/'
-
-    @property
-    def not_started(self):
-        return self.status == 'not_started'
-
-    @property
-    def queued(self):
-        return self.status == 'queued'
-
-    @property
-    def running(self):
-        return self.status == 'running'
-
-    @property
-    def success(self):
-        return self.status == 'success'
-
-    @property
-    def failure(self):
-        return self.status == 'failure'
-
-    @property
-    def stopped(self):
-        return self.status == 'stopped'
-
-    @property
-    def stale(self):
-        return self.status == 'stale'
-
-
-def uri_fingerprint(uri):
-    """
-    Return the SHA1 hex digest of `uri`
-    """
-    encoded_uri = uri.encode('utf-8')
-    return hashlib.sha1(encoded_uri).hexdigest()
-
-
-def query_scans(uri, api_url=SCANCODEIO_API_URL_PROJECTS, api_auth_headers=SCANCODEIO_AUTH_HEADERS, response_save_loc=''):
-    """
-    Return scan information for `uri` if `uri` has already been scanned by ScanCode.io
-    """
-    payload = {'name': uri_fingerprint(uri)}
-    response = requests.get(url=api_url, params=payload, headers=api_auth_headers)
-    response_json = response.json()
-    if response_save_loc:
-        with open(response_save_loc, 'w') as f:
-            json.dump(response_json, f)
-    if not response.ok:
-        response.raise_for_status()
-    results = response_json['results']
-    if results and len(results) == 1:
-        return results[0]
-
-
-def submit_scan(
-    uri,
-    package,
-    api_url=SCANCODEIO_API_URL_PROJECTS,
-    api_auth_headers=SCANCODEIO_AUTH_HEADERS,
-    response_save_loc=''
-):
-    """
-    Submit a scan request for `uri` to ScanCode.io and return a Scan object on
-    success. Raise an exception on error.
-    """
-    logger.debug('submit_scan: uri', uri, 'api_url:', api_url, 'api_auth_headers:', api_auth_headers)
-    package_name = package.name
-    package_version = package.version
-    uuid = uuid4()
-    uuid_str = str(uuid)
-    uuid_segments = uuid_str.split('-')
-    uuid_segment = uuid_segments[-1]
-
-    if package_version:
-        project_name = f'{package_name}-{package_version}-{uuid_segment}'
-    else:
-        project_name = f'{package.name}-{uuid_segment}'
-
-    request_args = {
-        'name': project_name,
-        'pipeline': 'scan_and_fingerprint_package',
-        'input_urls': [
-            uri
-        ],
-        'execute_now': True
-    }
-
-    response = requests.post(url=api_url, data=request_args, headers=api_auth_headers)
-    try:
-        response_json = response.json()
-    except json.decoder.JSONDecodeError as e:
-        msg = f'Error occured when decoding scan request response: {e}'
-        raise Exception(msg)
-    if response_save_loc:
-        with open(response_save_loc, 'w') as f:
-            json.dump(response_json, f)
-
-    if not response.ok:
-        if response.status_code == requests.codes.bad_request:
-            name = response_json.get('name')
-            if name and 'project with this name already exists.' in name:
-                query_results = query_scans(uri, api_url=api_url, api_auth_headers=api_auth_headers)
-                if query_results:
-                    scan = Scan.from_response(**query_results)
-            else:
-                response.raise_for_status()
-    else:
-        scan = Scan.from_response(**response_json)
-        uuid = scan.uuid
-        if not uuid:
-            msg = 'Failed to to submit scan UUID for URI: "{uri}".\n'.format(**locals())
-            msg += repr(response_json)
-            raise Exception(msg)
-    return scan
-
-
-def get_scan_url(scan_uuid, api_url=SCANCODEIO_API_URL_PROJECTS, suffix=''):
-    """
-    Return a scancode.io scan API URL built from the Scan UUID `scan_uuid` or
-    None. Return the basic URL to get scan request information. Optionally adds
-    a `suffix` (such as /data or /summary) to get scans data.
-
-    For example:
-        https://scancode.io/api/projects/b15f2dcb-46ef-43e1-b5e3-563871ce59cc/
-    """
-
-    base_url = api_url and api_url.rstrip('/') or ''
-    url = f'{base_url}/{scan_uuid}/{suffix}'
-    # scancode.io seems to demand a trailing slash
-    url = url.rstrip('/')
-    url = url + '/'
-    return url
-
-
-def _call_scan_get_api(
-        scan_uuid,
-        endpoint='',
-        api_url=SCANCODEIO_API_URL_PROJECTS,
-        api_auth_headers=SCANCODEIO_AUTH_HEADERS,
-        timeout=REQUEST_TIMEOUT,
-):
-    """
-    Send a get request to the scan API for `scan_uuid` and return response
-    mapping from a JSON response. Call either the plain scan enpoint or the data
-    or summary endpoints based on the value of the `endpoint `arg. Raise an
-    exception on error.
-    """
-    scan_url = get_scan_url(scan_uuid, api_url=api_url, suffix=endpoint)
-    try:
-        response = requests.get(url=scan_url, timeout=timeout, headers=api_auth_headers)
-        if not response.ok:
-            response.raise_for_status()
-    except Exception:
-        # Ensure that exceptions are passed up the call stack, so they can be
-        # caught when _call_scan_get_api is called by another function.
-        raise
-    return response.json()
-
-
-def _get_scan_info(
-    scan_uuid,
-    api_url=SCANCODEIO_API_URL_PROJECTS,
-    api_auth_headers=SCANCODEIO_AUTH_HEADERS,
-    timeout=REQUEST_TIMEOUT,
-    get_scan_info_save_loc=''
-):
-    """
-    Return a mapping of project info for `scan_uuid` fetched from ScanCode.io or None.
-    Raise an exception on error.
-    """
-    results = _call_scan_get_api(
-        scan_uuid,
-        endpoint='',
-        api_url=api_url,
-        api_auth_headers=api_auth_headers,
-        timeout=timeout
-    )
-    if get_scan_info_save_loc:
-        with open(get_scan_info_save_loc, 'w') as f:
-            json.dump(results, f)
-    return results
-
-
-def get_scan_info(
-    scan_uuid,
-    api_url=SCANCODEIO_API_URL_PROJECTS,
-    api_auth_headers=SCANCODEIO_AUTH_HEADERS,
-    timeout=REQUEST_TIMEOUT,
-    get_scan_info_save_loc=''
-):
-    """
-    Return a Scan object for `scan_uuid` fetched from ScanCode.io or None.
-    Raise an exception on error.
-    """
-    results = _get_scan_info(
-        scan_uuid=scan_uuid,
-        api_url=api_url,
-        api_auth_headers=api_auth_headers,
-        timeout=timeout,
-        get_scan_info_save_loc=get_scan_info_save_loc,
-    )
-    return Scan.from_response(**results)
-
-
-def get_scan_data(
-    scan_uuid,
-    api_url=SCANCODEIO_API_URL_PROJECTS,
-    api_auth_headers=SCANCODEIO_AUTH_HEADERS,
-    timeout=REQUEST_TIMEOUT,
-    get_scan_data_save_loc=''
-):
-    """
-    Return scan details data as a mapping for a `scan_uuid` fetched from
-    ScanCode.io or None. Raise an exception on error.
-    """
-    # FIXME: we should return a temp location instead
-    results = _call_scan_get_api(
-        scan_uuid,
-        endpoint='results',
-        api_url=api_url,
-        api_auth_headers=api_auth_headers,
-        timeout=timeout
-    )
-    if get_scan_data_save_loc:
-        with open(get_scan_data_save_loc, 'w') as f:
-            json.dump(results, f)
-    return results
-
-
-def get_scan_summary(
-    scan_uuid,
-    api_url=SCANCODEIO_API_URL_PROJECTS,
-    api_auth_headers=SCANCODEIO_AUTH_HEADERS,
-    timeout=REQUEST_TIMEOUT,
-    get_scan_data_save_loc=''
-):
-    """
-    Return scan summary data as a mapping for a `scan_uuid` fetched from
-    ScanCode.io or None. Raise an exception on error.
-    """
-    # FIXME: we should return a temp location instead
-    results = _call_scan_get_api(
-        scan_uuid,
-        endpoint='summary',
-        api_url=api_url,
-        api_auth_headers=api_auth_headers,
-        timeout=timeout
-    )
-    if get_scan_data_save_loc:
-        with open(get_scan_data_save_loc, 'w') as f:
-            json.dump(results, f)
-    return results
-
-
-class ScanningCommand(VerboseCommand):
-    """
-    Base command class for processing ScannableURIs.
-    """
-    # subclasses must override
-    logger = None
-
-    api_url = SCANCODEIO_API_URL_PROJECTS
-
-    api_auth_headers = SCANCODEIO_AUTH_HEADERS
-
-    def add_arguments(self, parser):
-        parser.add_argument(
-            '--exit-on-empty',
-            dest='exit_on_empty',
-            default=False,
-            action='store_true',
-            help='Do not loop forever. Exit when the queue is empty.')
-
-        parser.add_argument(
-            '--max-uris',
-            dest='max_uris',
-            default=0,
-            action='store',
-            help='Limit the number of Scannable URIs processed to a maximum number. '
-                 '0 means no limit. Used only for testing.')
-
-    def handle(self, *args, **options):
-        exit_on_empty = options.get('exit_on_empty')
-        max_uris = options.get('max_uris', 0)
-
-        uris_counter = self.process_scans(
-            exit_on_empty=exit_on_empty,
-            max_uris=max_uris,
-            # Pass options to allow subclasses to add their own options
-            options=options
-        )
-        self.stdout.write('Processed {} ScannableURI.'.format(uris_counter))
-
-    @classmethod
-    def process_scans(cls, exit_on_empty=False, max_uris=0, **kwargs):
-        """
-        Run an infinite scan processing loop. Return a processed URis count.
-
-        Get the next available candidate ScannableURI and request a scan from
-        ScanCode.io. Loops forever and sleeps a short while if there are no
-        ScannableURI left to scan.
-        """
-        uris_counter = 0
-        sleeping = False
-
-        while True:
-            # Wait before processing anything
-            time.sleep(3)
-
-            if cls.MUST_STOP:
-                cls.logger.info('Graceful exit of the scan processing loop.')
-                break
-
-            if max_uris and uris_counter >= max_uris:
-                cls.logger.info('max_uris requested reached: exiting scan processing loop.')
-                break
-
-            scannable_uri = cls.get_next_uri()
-
-            if not scannable_uri:
-                if exit_on_empty:
-                    cls.logger.info('exit-on-empty requested: No more scannable URIs, exiting...')
-                    break
-
-                # Only log a single message when we go to sleep
-                if not sleeping:
-                    sleeping = True
-                    cls.logger.info('No more scannable URIs, sleeping for at least {} seconds...'.format(SLEEP_WHEN_EMPTY))
-
-                time.sleep(SLEEP_WHEN_EMPTY)
-                continue
-
-            cls.logger.info('Processing scannable URI: {}'.format(scannable_uri))
-
-            cls.process_scan(scannable_uri, **kwargs)
-            uris_counter += 1
-            sleeping = False
-
-        return uris_counter
-
-    @classmethod
-    def get_next_uri(self):
-        """
-        Return a locked ScannableURI for processing.
-        Subclasses must implement
-
-        Typically something like:
-            with transaction.atomic():
-                scannable_uri = ScannableURI.objects.get_next_scannable()
-        """
-        pass
-
-    @classmethod
-    def process_scan(scannable_uri, **kwargs):
-        """
-        Process a single `scannable_uri` ScannableURI. Subclasses must implement.
-        If sucessfully processed the ScannableURI must be updated accordingly.
-        """
-        pass
diff --git a/packagedb/management/commands/create-user.py b/minecode/management/user_creation.py
similarity index 55%
rename from packagedb/management/commands/create-user.py
rename to minecode/management/user_creation.py
index 64d25cdc..3a3fe6f2 100644
--- a/packagedb/management/commands/create-user.py
+++ b/minecode/management/user_creation.py
@@ -1,24 +1,11 @@
-# SPDX-License-Identifier: Apache-2.0
-#
-# http://nexb.com and https://github.com/nexB/scancode.io
-# The ScanCode.io software is licensed under the Apache License version 2.0.
-# Data generated with ScanCode.io is provided as-is without warranties.
-# ScanCode is a trademark of nexB Inc.
-#
-# You may not use this software except in compliance with the License.
-# You may obtain a copy of the License at: http://apache.org/licenses/LICENSE-2.0
-# Unless required by applicable law or agreed to in writing, software distributed
-# under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
-# CONDITIONS OF ANY KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations under the License.
 #
-# Data Generated with ScanCode.io is provided on an "AS IS" BASIS, WITHOUT WARRANTIES
-# OR CONDITIONS OF ANY KIND, either express or implied. No content created from
-# ScanCode.io should be considered or used as legal advice. Consult an Attorney
-# for any legal advice.
+# Copyright (c) nexB Inc. and others. All rights reserved.
+# purldb is a trademark of nexB Inc.
+# SPDX-License-Identifier: Apache-2.0
+# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
+# See https://github.com/nexB/purldb for support or download.
+# See https://aboutcode.org for more information about nexB OSS projects.
 #
-# ScanCode.io is a free software code scanning tool from nexB Inc. and others.
-# Visit https://github.com/nexB/scancode.io for support and download.
 
 import getpass
 
@@ -31,8 +18,8 @@
 from rest_framework.authtoken.models import Token
 
 
-class Command(BaseCommand):
-    help = "Create a user and generate an API key for authentication."
+class CreateUserCommand(BaseCommand):
+    help = 'Create a user and generate an API key for authentication.'
     requires_migrations_checks = True
 
     def __init__(self, *args, **kwargs):
@@ -43,32 +30,42 @@ def __init__(self, *args, **kwargs):
         )
 
     def add_arguments(self, parser):
-        parser.add_argument("username", help="Specifies the username for the user.")
+        parser.add_argument('username', help='Specifies the username for the user.')
         parser.add_argument(
-            "--no-input",
-            action="store_false",
-            dest="interactive",
-            help="Do not prompt the user for input of any kind.",
+            '--no-input',
+            action='store_false',
+            dest='interactive',
+            help='Do not prompt the user for input of any kind.',
         )
 
     def handle(self, *args, **options):
-        username = options["username"]
+        username = options['username']
+        interactive = options['interactive']
+        verbosity = options['verbosity']
+        self.create_user(
+            username=username,
+            interactive=interactive,
+            verbosity=verbosity
+        )
 
+    def create_user(self, username, interactive, verbosity):
         error_msg = self._validate_username(username)
         if error_msg:
             raise CommandError(error_msg)
 
         password = None
-        if options["interactive"]:
+        if interactive:
             password = self.get_password_from_stdin(username)
 
         user = self.UserModel._default_manager.create_user(username, password=password)
         token, _ = Token._default_manager.get_or_create(user=user)
 
-        if options["verbosity"] >= 1:
-            msg = f"User {username} created with API key: {token.key}"
+        if verbosity >= 1:
+            msg = f'User {username} created with API key: {token.key}'
             self.stdout.write(msg, self.style.SUCCESS)
 
+        return user
+
     def get_password_from_stdin(self, username):
         # Validators, such as UserAttributeSimilarityValidator, depends on other user's
         # fields data for password validation.
@@ -79,21 +76,21 @@ def get_password_from_stdin(self, username):
         password = None
         while password is None:
             password1 = getpass.getpass()
-            password2 = getpass.getpass("Password (again): ")
+            password2 = getpass.getpass('Password (again): ')
             if password1 != password2:
                 self.stderr.write("Error: Your passwords didn't match.")
                 continue
-            if password1.strip() == "":
+            if password1.strip() == '':
                 self.stderr.write("Error: Blank passwords aren't allowed.")
                 continue
             try:
                 validate_password(password2, self.UserModel(**fake_user_data))
             except exceptions.ValidationError as err:
-                self.stderr.write("\n".join(err.messages))
+                self.stderr.write('\n'.join(err.messages))
                 response = input(
-                    "Bypass password validation and create user anyway? [y/N]: "
+                    'Bypass password validation and create user anyway? [y/N]: '
                 )
-                if response.lower() != "y":
+                if response.lower() != 'y':
                     continue
             password = password1
 
@@ -107,9 +104,9 @@ def _validate_username(self, username):
             except self.UserModel.DoesNotExist:
                 pass
             else:
-                return "Error: That username is already taken."
+                return 'Error: That username is already taken.'
 
         try:
             self.username_field.clean(username, None)
         except exceptions.ValidationError as e:
-            return "; ".join(e.messages)
+            return '; '.join(e.messages)
diff --git a/minecode/migrations/0032_remove_scannableuri_minecode_sc_scan_st_d6a459_idx_and_more.py b/minecode/migrations/0032_remove_scannableuri_minecode_sc_scan_st_d6a459_idx_and_more.py
new file mode 100644
index 00000000..937adaab
--- /dev/null
+++ b/minecode/migrations/0032_remove_scannableuri_minecode_sc_scan_st_d6a459_idx_and_more.py
@@ -0,0 +1,72 @@
+# Generated by Django 5.0.1 on 2024-03-07 00:59
+
+import uuid
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("minecode", "0031_importableuri"),
+        ("packagedb", "0083_delete_apiuser"),
+    ]
+
+    operations = [
+        migrations.RemoveIndex(
+            model_name="scannableuri",
+            name="minecode_sc_scan_st_d6a459_idx",
+        ),
+        migrations.RenameField(
+            model_name="scannableuri",
+            old_name="rescan_uri",
+            new_name="reindex_uri",
+        ),
+        migrations.AlterUniqueTogether(
+            name="scannableuri",
+            unique_together=set(),
+        ),
+        migrations.AddField(
+            model_name="scannableuri",
+            name="pipelines",
+            field=models.JSONField(
+                blank=True,
+                default=list,
+                editable=False,
+                help_text="A list of ScanCode.io pipeline names to be run for this scan",
+            ),
+        ),
+        migrations.AddField(
+            model_name="scannableuri",
+            name="scan_date",
+            field=models.DateTimeField(
+                blank=True,
+                db_index=True,
+                help_text="Timestamp set to the date when a scan was taken by a worker",
+                null=True,
+            ),
+        ),
+        migrations.AddField(
+            model_name="scannableuri",
+            name="uuid",
+            field=models.UUIDField(default=uuid.uuid4, null=True),
+        ),
+        migrations.AddIndex(
+            model_name="scannableuri",
+            index=models.Index(
+                fields=["scan_status", "scan_date"],
+                name="minecode_sc_scan_st_baab37_idx",
+            ),
+        ),
+        migrations.RemoveField(
+            model_name="scannableuri",
+            name="last_status_poll_date",
+        ),
+        migrations.RemoveField(
+            model_name="scannableuri",
+            name="scan_request_date",
+        ),
+        migrations.RemoveField(
+            model_name="scannableuri",
+            name="scan_uuid",
+        ),
+    ]
diff --git a/minecode/migrations/0033_scannableuri_populate_uuid.py b/minecode/migrations/0033_scannableuri_populate_uuid.py
new file mode 100644
index 00000000..e96b15a6
--- /dev/null
+++ b/minecode/migrations/0033_scannableuri_populate_uuid.py
@@ -0,0 +1,43 @@
+# Generated by Django 5.0.1 on 2024-02-16 00:16
+
+from django.db import migrations
+import uuid
+
+
+def populate_uuid(apps, schema_editor):
+    ScannableURI = apps.get_model("minecode", "ScannableURI")
+    unupdated = []
+    scannable_uris = ScannableURI.objects.all().iterator(chunk_size=5000)
+    for i, scannable_uri in enumerate(scannable_uris):
+        if  i > 0 and not i % 5000:
+            ScannableURI.objects.bulk_update(
+                objs=unupdated,
+                fields=[
+                    "uuid"
+                ]
+            )
+            unupdated = []
+        scannable_uri.uuid = uuid.uuid4()
+        unupdated.append(scannable_uri)
+
+    if unupdated:
+        ScannableURI.objects.bulk_update(
+            objs=unupdated,
+            fields=[
+                "uuid",
+            ]
+        )
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        (
+            "minecode",
+            "0032_remove_scannableuri_minecode_sc_scan_st_d6a459_idx_and_more",
+        ),
+    ]
+
+    operations = [
+        migrations.RunPython(populate_uuid, reverse_code=migrations.RunPython.noop),
+    ]
diff --git a/minecode/migrations/0034_scannableuri_alter_uuid_field.py b/minecode/migrations/0034_scannableuri_alter_uuid_field.py
new file mode 100644
index 00000000..28d88ace
--- /dev/null
+++ b/minecode/migrations/0034_scannableuri_alter_uuid_field.py
@@ -0,0 +1,19 @@
+# Generated by Django 5.0.1 on 2024-02-16 00:16
+
+from django.db import migrations, models
+import uuid
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("minecode", "0033_scannableuri_populate_uuid"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="scannableuri",
+            name="uuid",
+            field=models.UUIDField(default=uuid.uuid4, editable=False, unique=True),
+        ),
+    ]
diff --git a/minecode/migrations/0035_create_scan_queue_workers_group.py b/minecode/migrations/0035_create_scan_queue_workers_group.py
new file mode 100644
index 00000000..786e6017
--- /dev/null
+++ b/minecode/migrations/0035_create_scan_queue_workers_group.py
@@ -0,0 +1,39 @@
+# Generated by Django 5.0.1 on 2024-03-04 19:33
+
+from django.db import migrations
+
+
+from django.contrib.auth.models import Group, Permission
+from django.contrib.contenttypes.models import ContentType
+
+
+def create_scan_queue_workers_group_and_permissions(apps, schema_editor):
+    """
+    Create the `scan_workers` groups and set permissions for it
+    """
+    ScannableURI = apps.get_model('minecode', 'ScannableURI')
+    scan_queue_workers_group, _ = Group.objects.get_or_create(name='scan_queue_workers')
+    # Add permissions to `scan_queue_workers` group
+    scannable_uri_content_type = ContentType.objects.get_for_model(ScannableURI)
+    names_by_codenames = {
+        'change_scannableuri': 'Can change Scannable URI',
+        'view_scannableuri': 'Can view Scannable URI',
+    }
+    for codename, name in names_by_codenames.items():
+        permission, _ = Permission.objects.get_or_create(
+            codename=codename,
+            name=name,
+            content_type=scannable_uri_content_type
+        )
+        scan_queue_workers_group.permissions.add(permission)
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("minecode", "0034_scannableuri_alter_uuid_field"),
+    ]
+
+    operations = [
+        migrations.RunPython(create_scan_queue_workers_group_and_permissions, migrations.RunPython.noop),
+    ]
diff --git a/minecode/model_utils.py b/minecode/model_utils.py
index a600b104..f0c78b77 100644
--- a/minecode/model_utils.py
+++ b/minecode/model_utils.py
@@ -24,14 +24,29 @@
 logger.setLevel(logging.INFO)
 
 
-def add_package_to_scan_queue(package):
+# These are the list of default pipelines to run when we scan a Package for
+# indexing
+DEFAULT_PIPELINES = (
+    'scan_single_package',
+    'fingerprint_codebase',
+)
+
+
+def add_package_to_scan_queue(package, pipelines=DEFAULT_PIPELINES, reindex_uri=False, priority=0):
     """
-    Add a Package `package` to the scan queue
+    Add a Package `package` to the scan queue to run the list of provided `pipelines`
+
+    If `reindex_uri` is True, force rescanning of the package
     """
+    if not pipelines:
+        raise Exception('pipelines required to add package to scan queue')
     uri = package.download_url
     _, scannable_uri_created = ScannableURI.objects.get_or_create(
         uri=uri,
+        pipelines=pipelines,
         package=package,
+        reindex_uri=reindex_uri,
+        priority=priority,
     )
     if scannable_uri_created:
         logger.debug(' + Inserted ScannableURI\t: {}'.format(uri))
diff --git a/minecode/models.py b/minecode/models.py
index 3a6f046e..41f2caa3 100644
--- a/minecode/models.py
+++ b/minecode/models.py
@@ -7,7 +7,7 @@
 # See https://aboutcode.org for more information about nexB OSS projects.
 #
 
-
+import uuid
 from datetime import timedelta
 import logging
 import sys
@@ -649,27 +649,24 @@ class ScannableURI(BaseURI):
         - update the matching index for the PackageDB as needed with fingerprints from the scan
         - set status and timestamps as needed
     """
-    scan_request_date = models.DateTimeField(
-        null=True,
-        blank=True,
-        db_index=True,
-        help_text='Timestamp set to the date when a scan was requested.  Used to track scan status.',
+    uuid = models.UUIDField(
+        default=uuid.uuid4,
+        unique=True,
+        editable=False,
     )
 
-    last_status_poll_date = models.DateTimeField(
+    scan_date = models.DateTimeField(
         null=True,
         blank=True,
         db_index=True,
-        help_text='Timestamp set to the date of the last status poll. '
-                  'Used to track the scan polling.',
+        help_text='Timestamp set to the date when a scan was taken by a worker',
     )
 
-    scan_uuid = models.CharField(
-        max_length=36,
+    pipelines = models.JSONField(
+        default=list,
         blank=True,
-        null=True,
-        db_index=True,
-        help_text='UUID of a scan for this URI in ScanCode.io.',
+        editable=False,
+        help_text='A list of ScanCode.io pipeline names to be run for this scan',
     )
 
     SCAN_NEW = 0
@@ -694,6 +691,12 @@ class ScannableURI(BaseURI):
 
     SCAN_STATUSES_BY_CODE = dict(SCAN_STATUS_CHOICES)
 
+    SCAN_STATUS_CODES_BY_SCAN_STATUS = {
+        status: code
+        for code, status
+        in SCAN_STATUS_CHOICES
+    }
+
     scan_status = models.IntegerField(
         default=SCAN_NEW,
         choices=SCAN_STATUS_CHOICES,
@@ -701,21 +704,13 @@ class ScannableURI(BaseURI):
         help_text='Status of the scan for this URI.',
     )
 
-    rescan_uri = models.BooleanField(
+    reindex_uri = models.BooleanField(
         default=False,
         null=True,
         blank=True,
         help_text='Flag indicating whether or not this URI should be rescanned and reindexed.',
     )
 
-    scan_uuid = models.CharField(
-        max_length=36,
-        blank=True,
-        null=True,
-        db_index=True,
-        help_text='UUID of a scan for this URI in ScanCode.io.',
-    )
-
     scan_error = models.TextField(
         null=True,
         blank=True,
@@ -739,15 +734,21 @@ class ScannableURI(BaseURI):
 
     class Meta:
         verbose_name = 'Scannable URI'
-        unique_together = ['canonical', 'scan_uuid']
 
         indexes = [
             # to get the scannables
             models.Index(
-                fields=['scan_status', 'scan_request_date', 'last_status_poll_date', ]),
+                fields=[
+                    'scan_status',
+                    'scan_date',
+                ]
+            ),
             # ordered by for the main queue query e.g. '-priority'
             models.Index(
-                fields=['-priority'])
+                fields=[
+                    '-priority'
+                ]
+            )
         ]
 
     def save(self, *args, **kwargs):
@@ -759,19 +760,6 @@ def save(self, *args, **kwargs):
         self.normalize_fields()
         super(ScannableURI, self).save(*args, **kwargs)
 
-    def rescan(self):
-        """
-        Reset fields such that a ScannableURI can be sent off for scanning again
-        """
-        self.rescan_uri = True
-        self.scan_status = ScannableURI.SCAN_NEW
-        self.scan_error = None
-        self.index_error = None
-        self.scan_uuid = None
-        self.scan_request_date = None
-        self.priority = 100
-        self.save()
-
 
 # TODO: Use the QuerySet.as_manager() for more flexibility and chaining.
 class PriorityResourceURIManager(models.Manager):
diff --git a/minecode/permissions.py b/minecode/permissions.py
new file mode 100644
index 00000000..83815cea
--- /dev/null
+++ b/minecode/permissions.py
@@ -0,0 +1,9 @@
+from rest_framework import permissions
+
+
+class IsScanQueueWorkerAPIUser(permissions.BasePermission):
+    """
+    Allow access to a user who is a part of the `scan_queue_workers` group
+    """
+    def has_permission(self, request, view):
+        return request.user.groups.filter(name='scan_queue_workers').exists()
diff --git a/minecode/utils.py b/minecode/utils.py
index 7ec0e736..7034ea52 100644
--- a/minecode/utils.py
+++ b/minecode/utils.py
@@ -12,6 +12,7 @@
 import logging
 import os
 import tempfile
+import uuid
 
 from django.conf import settings
 from django.utils.encoding import force_str
@@ -382,3 +383,11 @@ def form_vcs_url(vcs_tool, vcs_url, revision_tag_or_branch=None, sub_path=None):
         if sub_path:
             vcs_url = '#'.join(str(v) for v in [vcs_url, sub_path])
     return vcs_url
+
+
+def validate_uuid(uuid_string):
+    try:
+        val = uuid.UUID(uuid_string)
+    except ValueError:
+        return False
+    return str(val).lower() == uuid_string.lower()
diff --git a/minecode/visitors/maven.py b/minecode/visitors/maven.py
index d945d870..ba8a209b 100644
--- a/minecode/visitors/maven.py
+++ b/minecode/visitors/maven.py
@@ -8,19 +8,20 @@
 #
 
 from collections import namedtuple
+from typing import Dict
+from urllib.parse import urlparse
 import gzip
 import hashlib
 import io
 import json
 import logging
+import os
 import re
-from typing import Dict
-from urllib.parse import urlparse
 
-import arrow
-import requests
 from bs4 import BeautifulSoup
 from dateutil import tz
+import arrow
+import requests
 
 from jawa.util.utf import decode_modified_utf8
 import javaproperties
@@ -29,12 +30,11 @@
 from packagedcode.maven import build_filename
 from packagedcode.maven import build_url
 from packagedcode.maven import get_urls
-from packagedcode.maven import _parse
 from packagedcode.maven import get_maven_pom
-from packageurl import PackageURL
+from packagedcode.maven import _parse
 
-from minecode import seed
 from minecode import priority_router
+from minecode import seed
 from minecode import visit_router
 from minecode.visitors import java_stream
 from minecode.visitors import HttpVisitor
@@ -44,6 +44,7 @@
 from packagedb.models import make_relationship
 from packagedb.models import PackageContentType
 from packagedb.models import PackageRelation
+from packagedb.models import make_relationship
 
 """
 This module handles the Maven repositories such as central and other
@@ -236,7 +237,7 @@ def merge_ancestors(ancestor_pom_texts, package):
             datasource_id='maven_pom',
             package_type='maven',
             primary_language='Java',
-            text=ancestor_pom_text
+            text=ancestor_pom_text,
         )
         package = merge_parent(package, ancestor_package)
     return package
@@ -248,14 +249,13 @@ def map_maven_package(package_url, package_content):
 
     Return an error string if errors have occured in the process.
     """
-    from minecode.model_utils import add_package_to_scan_queue
-    from minecode.model_utils import merge_or_create_package
+    from minecode.model_utils import add_package_to_scan_queue, merge_or_create_package
 
     db_package = None
     error = ''
 
-    if "repository_url" in package_url.qualifiers:
-        base_url = package_url.qualifiers["repository_url"]
+    if 'repository_url' in package_url.qualifiers:
+        base_url = package_url.qualifiers['repository_url']
     else:
         base_url = MAVEN_BASE_URL
 
@@ -280,11 +280,7 @@ def map_maven_package(package_url, package_content):
         base_url=base_url,
     )
     ancestor_pom_texts = get_ancestry(pom_text=pom_text, base_url=base_url)
-    package = merge_ancestors(
-        ancestor_pom_texts=ancestor_pom_texts,
-        package=package
-    )
-
+    package = merge_ancestors(ancestor_pom_texts=ancestor_pom_texts, package=package)
 
     urls = get_urls(
         namespace=package_url.namespace,
@@ -333,18 +329,14 @@ def map_maven_binary_and_source(package_url):
     Return an error string for errors that occur, or empty string if there is no error.
     """
     error = ''
-    package, emsg = map_maven_package(
-        package_url,
-        PackageContentType.BINARY
-    )
+    package, emsg = map_maven_package(package_url, PackageContentType.BINARY)
     if emsg:
         error += emsg
 
     source_package_url = package_url
     source_package_url.qualifiers['classifier'] = 'sources'
     source_package, emsg = map_maven_package(
-        source_package_url,
-        PackageContentType.SOURCE_ARCHIVE
+        source_package_url, PackageContentType.SOURCE_ARCHIVE
     )
     if emsg:
         error += emsg
@@ -353,7 +345,7 @@ def map_maven_binary_and_source(package_url):
         make_relationship(
             from_package=source_package,
             to_package=package,
-            relationship=PackageRelation.Relationship.SOURCE_PACKAGE
+            relationship=PackageRelation.Relationship.SOURCE_PACKAGE,
         )
 
     return error
@@ -381,7 +373,7 @@ def map_maven_packages(package_url):
             type='maven',
             namespace=listing.get('g'),
             name=listing.get('a'),
-            version=listing.get('v')
+            version=listing.get('v'),
         )
         emsg = map_maven_binary_and_source(purl)
         if emsg:
@@ -472,17 +464,15 @@ def check_if_package_version_page(links, **kwargs):
     """
     Return True if `links` contains pom files and has no directories
     """
-    return (
-        check_if_page_has_pom_files(links=links)
-        and not check_if_page_has_directories(links=links)
-    )
+    return check_if_page_has_pom_files(
+        links=links
+    ) and not check_if_page_has_directories(links=links)
 
 
 def check_if_package_page(links, **kwargs):
-    return (
-        check_if_file_name_is_linked_on_page(file_name='maven-metadata.xml', links=links)
-        and not check_if_page_has_pom_files(links=links)
-    )
+    return check_if_file_name_is_linked_on_page(
+        file_name='maven-metadata.xml', links=links
+    ) and not check_if_page_has_pom_files(links=links)
 
 
 def check_if_maven_root(links, **kwargs):
@@ -490,7 +480,9 @@ def check_if_maven_root(links, **kwargs):
     Return True if "archetype-catalog.xml" is in `links`, as the root of a Maven
     repo contains "archetype-catalog.xml".
     """
-    return check_if_file_name_is_linked_on_page(file_name='archetype-catalog.xml', links=links)
+    return check_if_file_name_is_linked_on_page(
+        file_name='archetype-catalog.xml', links=links
+    )
 
 
 def check_on_page(url, checker):
@@ -551,7 +543,7 @@ def get_maven_root(url):
     """
     scheme, netloc, path_segments = url_parts(url)
     for i in range(len(path_segments)):
-        segments = path_segments[:i+1]
+        segments = path_segments[: i + 1]
         url_segment = create_url(scheme, netloc, segments)
         if is_maven_root(url_segment):
             return url_segment
@@ -583,7 +575,7 @@ def determine_namespace_name_version_from_url(url, root_url=None):
     package_version = ''
     for i in range(len(remaining_path_segments)):
         segment = remaining_path_segments[i]
-        segments = remaining_path_segments[:i+1]
+        segments = remaining_path_segments[: i + 1]
         path = '/'.join(segments)
         url_segment = f'{root_url}/{path}'
         if is_package_page(url_segment):
@@ -601,6 +593,7 @@ def add_to_import_queue(url, root_url):
     Create ImportableURI for the Maven repo package page at `url`.
     """
     from minecode.models import ImportableURI
+
     data = None
     response = requests.get(url)
     if response:
@@ -699,9 +692,7 @@ def get_directory_links(url):
     response = requests.get(url)
     if response:
         timestamps_by_directory_links = create_absolute_urls_for_links(
-            response.text,
-            url=url,
-            filter=filter_only_directories
+            response.text, url=url, filter=filter_only_directories
         )
     return timestamps_by_directory_links
 
@@ -714,9 +705,7 @@ def get_artifact_links(url):
     response = requests.get(url)
     if response:
         timestamps_by_artifact_links = create_absolute_urls_for_links(
-            response.text,
-            url=url,
-            filter=filter_for_artifacts
+            response.text, url=url, filter=filter_for_artifacts
         )
     return timestamps_by_artifact_links
 
@@ -755,7 +744,9 @@ def get_artifact_sha1(artifact_url):
     return sha1
 
 
-def get_classifier_from_artifact_url(artifact_url, package_version_page_url, package_name, package_version):
+def get_classifier_from_artifact_url(
+    artifact_url, package_version_page_url, package_name, package_version
+):
     """
     Return the classifier from a Maven artifact URL `artifact_url`, otherwise
     return None if a classifier cannot be determined from `artifact_url`
@@ -1555,19 +1546,19 @@ def _artifact_stats(location):
 
     print('Top packaging:')
     for n, c in pom_packs.most_common():
-        print(n, ":", c)
+        print(n, ':', c)
 
     print('Top classifiers:')
     for n, c in pom_classifs.most_common():
-        print(n, ":", c)
+        print(n, ':', c)
 
     print('Top extensions:')
     for n, c in pom_extensions.most_common():
-        print(n, ":", c)
+        print(n, ':', c)
 
     print('Top Combos: packaging, classifier, extension')
     for n, c in combos.most_common():
-        print(n, ":", c)
+        print(n, ':', c)
 
     """
     Latest stats on 2017-08-07:
diff --git a/minecode/visitors/npm.py b/minecode/visitors/npm.py
index 26fca883..3fdceddf 100644
--- a/minecode/visitors/npm.py
+++ b/minecode/visitors/npm.py
@@ -7,20 +7,18 @@
 # See https://aboutcode.org for more information about nexB OSS projects.
 #
 
-
 import logging
 import json
-import requests
-
-from packageurl import PackageURL
 
 from packagedcode.npm import npm_api_url
 from packagedcode.npm import split_scoped_package_name
 from packagedcode.npm import NpmPackageJsonHandler
+from packageurl import PackageURL
+import requests
 
 from minecode import seed
-from minecode import visit_router
 from minecode import priority_router
+from minecode import visit_router
 from minecode.visitors import NonPersistentHttpVisitor
 from minecode.visitors import URI
 from packagedb.models import PackageContentType
@@ -139,10 +137,10 @@ def map_npm_package(package_url):
     from minecode.model_utils import merge_or_create_package
 
     package_json = get_package_json(
-        namespace = package_url.namespace,
+        namespace=package_url.namespace,
         name=package_url.name,
         version=package_url.version,
-        )
+    )
 
     if not package_json:
         error = f'Package does not exist on npmjs: {package_url}'
diff --git a/packagedb/api.py b/packagedb/api.py
index 82ed5461..20f9a269 100644
--- a/packagedb/api.py
+++ b/packagedb/api.py
@@ -426,7 +426,7 @@ def reindex_package(self, request, *args, **kwargs):
         Reindex this package instance
         """
         package = self.get_object()
-        package.rescan()
+        package.reindex()
         data = {
             'status': f'{package.package_url} has been queued for reindexing'
         }
@@ -557,7 +557,7 @@ class PackageSetViewSet(viewsets.ReadOnlyModelViewSet):
 class PackageWatchViewSet(CreateListRetrieveUpdateViewSetMixin):
     """
     Take a `purl` and periodically watch for the new version of the package.
-    Add the new package version to the scan queue. 
+    Add the new package version to the scan queue.
     Default watch interval is 7 days.
     """
     queryset = PackageWatch.objects.get_queryset().order_by('-id')
@@ -579,7 +579,7 @@ class CollectViewSet(viewsets.ViewSet):
 
     If the package does not exist, we will fetch the Package data and return
     it in the same request.
-    
+
     **Note:** Use `Index packages` for bulk indexing/reindexing of packages.
     """
     serializer_class=None
@@ -631,7 +631,7 @@ def list(self, request, format=None):
 
         serializer = PackageAPISerializer(packages, many=True, context={'request': request})
         return Response(serializer.data)
-    
+
     @extend_schema(
         request=IndexPackagesSerializer,
         responses={
@@ -705,7 +705,7 @@ def index_packages(self, request, *args, **kwargs):
         def _reindex_package(package, reindexed_packages):
             if package in reindexed_packages:
                 return
-            package.rescan()
+            package.reindex()
             reindexed_packages.append(package)
 
         serializer = self.serializer_class(data=request.data)
@@ -782,17 +782,17 @@ def _reindex_package(package, reindexed_packages):
 
 class PurlValidateViewSet(viewsets.ViewSet):
     """
-    Take a `purl` and check whether it's valid PackageURL or not.  
-    Optionally set `check_existence` to true to check whether the package exists in real world. 
-    
-    **Note:** As of now `check_existence` only supports `cargo`, `composer`, `deb`, 
+    Take a `purl` and check whether it's valid PackageURL or not.
+    Optionally set `check_existence` to true to check whether the package exists in real world.
+
+    **Note:** As of now `check_existence` only supports `cargo`, `composer`, `deb`,
     `gem`, `golang`, `hex`, `maven`, `npm`, `nuget` and `pypi` ecosystems.
 
     **Example request:**
             ```
             GET /api/validate/?purl=pkg:npm/foobar@12.3.1&check_existence=false
             ```
-    
+
     Response contains:
 
     - valid
@@ -801,7 +801,7 @@ class PurlValidateViewSet(viewsets.ViewSet):
         - True, if input PURL exists in real world and `check_existence` flag is enabled.
     """
     serializer_class = PurlValidateSerializer
-    
+
     def get_view_name(self):
         return 'Validate PURL'
 
@@ -876,7 +876,7 @@ def list(self, request):
                         response['exists'] = True
                 else:
                     unsupported_ecosystem = True
-            
+
             if response['exists']:
                 response["message"] = message_valid_and_exists
             elif unsupported_ecosystem:
@@ -884,7 +884,7 @@ def list(self, request):
                 response["message"] = message_valid_but_package_type_not_supported
             else:
                 response["message"] =message_valid_but_does_not_exist
-        
+
         serializer = PurlValidateResponseSerializer(response, context={'request': request})
         return Response(serializer.data)
 
@@ -1015,7 +1015,7 @@ def get_all_versions_plain(purl: PackageURL):
 
 def get_all_versions(purl):
     """
-    Return all the versions available for the given purls as 
+    Return all the versions available for the given purls as
     proper Version objects from `univers`.
     """
     all_versions = get_all_versions_plain(purl)
diff --git a/packagedb/management/commands/create_source_repo_packages.py b/packagedb/management/commands/create_source_repo_packages.py
index 3cde76e1..70617278 100644
--- a/packagedb/management/commands/create_source_repo_packages.py
+++ b/packagedb/management/commands/create_source_repo_packages.py
@@ -9,7 +9,6 @@
 
 import logging
 import sys
-from uuid import uuid4
 
 import openpyxl
 from packageurl.contrib.django.utils import purl_to_lookups
diff --git a/packagedb/migrations/0084_alter_resource_file_type_alter_resource_mime_type_and_more.py b/packagedb/migrations/0084_alter_resource_file_type_alter_resource_mime_type_and_more.py
new file mode 100644
index 00000000..952ae294
--- /dev/null
+++ b/packagedb/migrations/0084_alter_resource_file_type_alter_resource_mime_type_and_more.py
@@ -0,0 +1,43 @@
+# Generated by Django 5.0.1 on 2024-03-16 00:44
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("packagedb", "0083_delete_apiuser"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="resource",
+            name="file_type",
+            field=models.CharField(
+                blank=True,
+                help_text="Descriptive file type for this resource.",
+                max_length=1024,
+                null=True,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="resource",
+            name="mime_type",
+            field=models.CharField(
+                blank=True,
+                help_text="MIME type (aka. media type) for this resource. See https://en.wikipedia.org/wiki/Media_type",
+                max_length=100,
+                null=True,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="resource",
+            name="programming_language",
+            field=models.CharField(
+                blank=True,
+                help_text="Programming language of this resource if this is a code file.",
+                max_length=50,
+                null=True,
+            ),
+        ),
+    ]
diff --git a/packagedb/models.py b/packagedb/models.py
index 93c1690e..ac097ed3 100644
--- a/packagedb/models.py
+++ b/packagedb/models.py
@@ -620,24 +620,15 @@ def get_latest_version(self):
         if sorted_versions:
             return sorted_versions[-1]
 
-    # TODO: Should this be called `reindex` in this context?
-    def rescan(self):
+    def reindex(self):
         """
-        Trigger another scan of this Package, where the URI at `download_url` is
-        sent to scancode.io for a scan. The fingerprints and Resources associated with this
-        Package are deleted and recreated from the updated scan data.
+        Trigger another scan of this Package, where a new ScannableURI is
+        created for this Package. The fingerprints and Resources associated with
+        this Package are deleted and recreated from the updated scan data.
         """
-        from minecode.models import ScannableURI
+        from minecode.model_utils import add_package_to_scan_queue
 
-        # TODO: Consider sending a new scan request instead of reusing the
-        # existing one
-        try:
-            scannable_uri = ScannableURI.objects.get(package=self)
-        except ScannableURI.DoesNotExist:
-            scannable_uri = None
-
-        if scannable_uri:
-            scannable_uri.rescan()
+        add_package_to_scan_queue(self, reindex_uri=True, priority=100)
 
     def update_fields(self, save=False, **values_by_fields):
         """
@@ -803,6 +794,7 @@ def update_fields(self, save=False, **values_by_fields):
             self.append_to_history(
                 'Package field values have been updated.',
                 data=data,
+                save=save,
             )
             updated_fields.append('history')
 
@@ -813,9 +805,6 @@ def update_fields(self, save=False, **values_by_fields):
             # Deduplicate field names
             updated_fields = list(set(updated_fields))
 
-        if save:
-            self.save()
-
         return self, updated_fields
 
 
@@ -975,6 +964,7 @@ class AbstractResource(models.Model):
     mime_type = models.CharField(
         max_length=100,
         blank=True,
+        null=True,
         help_text=_(
             "MIME type (aka. media type) for this resource. "
             "See https://en.wikipedia.org/wiki/Media_type"
@@ -984,12 +974,14 @@ class AbstractResource(models.Model):
     file_type = models.CharField(
         max_length=1024,
         blank=True,
+        null=True,
         help_text=_("Descriptive file type for this resource."),
     )
 
     programming_language = models.CharField(
         max_length=50,
         blank=True,
+        null=True,
         help_text=_("Programming language of this resource if this is a code file."),
     )
 
diff --git a/purldb_project/urls.py b/purldb_project/urls.py
index 80105988..f9d9c460 100644
--- a/purldb_project/urls.py
+++ b/purldb_project/urls.py
@@ -24,6 +24,7 @@
 from matchcode.api import ExactFileIndexViewSet
 from matchcode.api import ExactPackageArchiveIndexViewSet
 from minecode.api import PriorityResourceURIViewSet
+from minecode.api import ScannableURIViewSet
 from packagedb.api import PurlValidateViewSet
 from packagedb.to_purl import api_to_purl_router
 from packagedb.from_purl import api_from_purl_router 
@@ -45,7 +46,7 @@
 api_router.register('validate', PurlValidateViewSet, 'validate')
 api_router.register('collect', CollectViewSet, 'collect')
 api_router.register('watch',PackageWatchViewSet)
-
+api_router.register('scan_queue', ScannableURIViewSet)
 
 urlpatterns = [
     path(


Test patch:
diff --git a/minecode/tests/create_scanning_fixtures.py b/minecode/tests/create_scanning_fixtures.py
deleted file mode 100644
index 831ee038..00000000
--- a/minecode/tests/create_scanning_fixtures.py
+++ /dev/null
@@ -1,60 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) nexB Inc. and others. All rights reserved.
-# purldb is a trademark of nexB Inc.
-# SPDX-License-Identifier: Apache-2.0
-# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
-# See https://github.com/nexB/purldb for support or download.
-# See https://aboutcode.org for more information about nexB OSS projects.
-#
-
-import os
-
-from django.test import TestCase
-
-from minecode.models import ScannableURI
-from packagedb.models import Package
-from minecode.management import scanning
-from minecode.utils_test import JsonBasedTesting
-from minecode.management.commands.request_scans import Command as RequestScansCommand
-from minecode.management.commands.process_scans import Command as ProcessScansCommand
-
-
-class ScanCodeIOAPIHelperFunctionTest(JsonBasedTesting, TestCase):
-    test_data_dir = os.path.join(os.path.dirname(__file__), 'testfiles')
-
-    def setUp(self):
-        self.package1, _ = Package.objects.get_or_create(
-            download_url='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-            type='maven',
-            namespace='',
-            name='wagon-api',
-            version='20040705.181715',
-        )
-        self.scannable_uri1, _ = ScannableURI.objects.get_or_create(
-            uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-            package=self.package1
-        )
-
-    def generate_api_responses(self):
-        scan_request_response_loc = self.get_test_loc('scancodeio/scan_request_response.json')
-        RequestScansCommand.process_scan(
-            self.scannable_uri1,
-            response_save_loc=scan_request_response_loc,
-            options={}
-        )
-        get_scan_info_save_loc = self.get_test_loc('scancodeio/get_scan_info.json')
-        get_scan_data_save_loc = self.get_test_loc('scancodeio/get_scan_data.json')
-        ProcessScansCommand.process_scan(
-            self.scannable_uri1,
-            get_scan_info_save_loc=get_scan_info_save_loc,
-            get_scan_data_save_loc=get_scan_data_save_loc
-        )
-        scan_exists_for_uri_save_loc = self.get_test_loc('scancodeio/scan_exists_for_uri.json')
-        RequestScansCommand.process_scan(
-            self.scannable_uri1,
-            response_save_loc=scan_exists_for_uri_save_loc,
-            options={}
-        )
-        scan_request_lookup_loc = self.get_test_loc('scancodeio/scan_request_lookup.json')
-        response = scanning.query_scans(uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar', response_save_loc=scan_request_lookup_loc)
diff --git a/minecode/tests/test_api.py b/minecode/tests/test_api.py
new file mode 100644
index 00000000..ea9aec36
--- /dev/null
+++ b/minecode/tests/test_api.py
@@ -0,0 +1,265 @@
+#
+# Copyright (c) nexB Inc. and others. All rights reserved.
+# purldb is a trademark of nexB Inc.
+# SPDX-License-Identifier: Apache-2.0
+# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
+# See https://github.com/nexB/purldb for support or download.
+# See https://aboutcode.org for more information about nexB OSS projects.
+#
+
+import json
+import os
+
+from django.contrib.auth.models import Group, User
+from django.test import TestCase
+from rest_framework import status
+from rest_framework.test import APIClient
+
+from minecode.models import ScannableURI
+from minecode.utils_test import JsonBasedTesting
+from packagedb.models import Package, Resource
+
+
+class ScannableURIAPITestCase(JsonBasedTesting, TestCase):
+    test_data_dir = os.path.join(os.path.dirname(__file__), 'testfiles')
+
+    def setUp(self):
+        self.scan_queue_worker_user = User.objects.create_user(
+            username="username",
+            email="e@mail.com",
+            password="secret"
+        )
+        scan_queue_workers_group, _ = Group.objects.get_or_create(name='scan_queue_workers')
+        scan_queue_workers_group.user_set.add(self.scan_queue_worker_user)
+        self.scan_queue_worker_auth = f"Token {self.scan_queue_worker_user.auth_token.key}"
+        self.scan_queue_worker_client = APIClient(enforce_csrf_checks=True)
+        self.scan_queue_worker_client.credentials(HTTP_AUTHORIZATION=self.scan_queue_worker_auth)
+
+        # create a staff user
+        self.staff_user = User.objects.create_user(
+            username="staff_username",
+            email="staff_e@mail.com",
+            password="secret",
+            is_staff=True
+        )
+        self.staff_auth = f"Token {self.staff_user.auth_token.key}"
+        self.staff_client = APIClient(enforce_csrf_checks=True)
+        self.staff_client.credentials(HTTP_AUTHORIZATION=self.staff_auth)
+
+        # create a regular user
+        self.regular_user = User.objects.create_user(
+            username="regular_username",
+            email="regular_e@mail.com",
+            password="secret",
+        )
+        self.regular_auth = f"Token {self.regular_user.auth_token.key}"
+        self.regular_client = APIClient(enforce_csrf_checks=True)
+        self.regular_client.credentials(HTTP_AUTHORIZATION=self.regular_auth)
+
+        self.anonymous_client = APIClient()
+
+        self.package1 = Package.objects.create(
+            download_url='https://test-url.com/package1.tar.gz',
+            type='type1',
+            name='name1',
+            version='1.0',
+        )
+        self.scannable_uri1 = ScannableURI.objects.create(
+            uri='https://test-url.com/package1.tar.gz',
+            package=self.package1
+        )
+
+        self.package2 = Package.objects.create(
+            download_url='https://test-url.com/package2.tar.gz',
+            type='type2',
+            name='name2',
+            version='2.0',
+        )
+        self.scannable_uri2 = ScannableURI.objects.create(
+            uri='https://test-url.com/package2.tar.gz',
+            package=self.package2
+        )
+
+        self.package3 = Package.objects.create(
+            download_url='https://test-url.com/package3.tar.gz',
+            type='type3',
+            name='name3',
+            version='3.0',
+        )
+        self.scannable_uri3 = ScannableURI.objects.create(
+            uri='https://test-url.com/package3.tar.gz',
+            package=self.package3
+        )
+
+    def test_api_scannable_uri_permissions(self):
+        response = self.anonymous_client.get('/api/scan_queue/')
+        self.assertEqual(response.status_code, status.HTTP_401_UNAUTHORIZED)
+
+        response = self.anonymous_client.get('/api/scan_queue/get_next_download_url/')
+        self.assertEqual(response.status_code, status.HTTP_401_UNAUTHORIZED)
+
+        response = self.anonymous_client.post('/api/scan_queue/update_status/')
+        self.assertEqual(response.status_code, status.HTTP_401_UNAUTHORIZED)
+
+        response = self.regular_client.get('/api/scan_queue/')
+        self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
+
+        response = self.regular_client.get('/api/scan_queue/get_next_download_url/')
+        self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
+
+        response = self.regular_client.post('/api/scan_queue/update_status/')
+        self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
+
+    def test_api_scannable_uri_list_endpoint(self):
+        response = self.scan_queue_worker_client.get('/api/scan_queue/')
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(3, response.data.get('count'))
+
+        response = self.staff_client.get('/api/scan_queue/')
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(3, response.data.get('count'))
+
+    def test_api_scannable_uri_get_next_download_url(self):
+        response = self.scan_queue_worker_client.get('/api/scan_queue/get_next_download_url/')
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(response.data.get('scannable_uri_uuid'), self.scannable_uri1.uuid)
+        self.assertEqual(response.data.get('download_url'), self.scannable_uri1.uri)
+
+        response = self.scan_queue_worker_client.get('/api/scan_queue/get_next_download_url/')
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(response.data.get('scannable_uri_uuid'), self.scannable_uri2.uuid)
+        self.assertEqual(response.data.get('download_url'), self.scannable_uri2.uri)
+
+        response = self.scan_queue_worker_client.get('/api/scan_queue/get_next_download_url/')
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(response.data.get('scannable_uri_uuid'), self.scannable_uri3.uuid)
+        self.assertEqual(response.data.get('download_url'), self.scannable_uri3.uri)
+
+        response = self.scan_queue_worker_client.get('/api/scan_queue/get_next_download_url/')
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(response.data.get('scannable_uri_uuid'), '')
+        self.assertEqual(response.data.get('download_url'), '')
+
+        response = self.staff_client.get('/api/scan_queue/get_next_download_url/')
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(response.data.get('scannable_uri_uuid'), '')
+        self.assertEqual(response.data.get('download_url'), '')
+
+    def test_api_scannable_uri_update_status(self):
+        self.assertEqual(ScannableURI.SCAN_NEW, self.scannable_uri1.scan_status)
+        data = {
+            "scannable_uri_uuid": self.scannable_uri1.uuid,
+            "scan_status": 'failed',
+            'scan_log': 'scan_log',
+        }
+        response = self.scan_queue_worker_client.post('/api/scan_queue/update_status/', data=data)
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.scannable_uri1.refresh_from_db()
+        self.assertEqual(ScannableURI.SCAN_FAILED, self.scannable_uri1.scan_status)
+        self.assertEqual('scan_log', self.scannable_uri1.scan_error)
+
+        self.assertFalse(self.package2.md5)
+        self.assertFalse(self.package2.sha1)
+        self.assertFalse(self.package2.sha256)
+        self.assertFalse(self.package2.sha512)
+        self.assertFalse(self.package2.size)
+        self.assertFalse(self.package2.declared_license_expression)
+        self.assertFalse(self.package2.copyright)
+        self.assertEqual(0, Resource.objects.all().count())
+        scan_file_location = self.get_test_loc('scancodeio/get_scan_data.json')
+        summary_file_location = self.get_test_loc('scancodeio/scan_summary_response.json')
+        project_extra_data = {
+            'md5': 'md5',
+            'sha1': 'sha1',
+            'sha256': 'sha256',
+            'sha512': 'sha512',
+            'size': 100,
+        }
+        with open(scan_file_location) as scan_file:
+            with open(summary_file_location) as summary_file:
+                data = {
+                    "scannable_uri_uuid": self.scannable_uri2.uuid,
+                    "scan_status": 'scanned',
+                    'project_extra_data': json.dumps(project_extra_data),
+                    'scan_results_file': scan_file,
+                    'scan_summary_file': summary_file,
+                }
+                response = self.scan_queue_worker_client.post('/api/scan_queue/update_status/', data=data)
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.scannable_uri2.refresh_from_db()
+        self.assertEqual(ScannableURI.SCAN_INDEXED, self.scannable_uri2.scan_status)
+        self.package2.refresh_from_db()
+        self.assertEqual('md5', self.package2.md5)
+        self.assertEqual('sha1', self.package2.sha1)
+        self.assertEqual('sha256', self.package2.sha256)
+        self.assertEqual('sha512', self.package2.sha512)
+        self.assertEqual(100, self.package2.size)
+        self.assertEqual('apache-2.0', self.package2.declared_license_expression)
+        self.assertEqual('Copyright (c) Apache Software Foundation', self.package2.copyright)
+        self.assertFalse(self.scannable_uri2.scan_error)
+        self.assertEqual(64, Resource.objects.all().count())
+
+        data = {}
+        response = self.scan_queue_worker_client.post('/api/scan_queue/update_status/', data=data)
+        expected_response = {'error': 'missing scannable_uri_uuid'}
+        self.assertEqual(expected_response, response.data)
+
+        data = {
+            'scannable_uri_uuid': self.scannable_uri2.uuid,
+            'scan_status': ''
+        }
+        response = self.scan_queue_worker_client.post('/api/scan_queue/update_status/', data=data)
+        expected_response = {'error': 'missing scan_status'}
+        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
+        self.assertEqual(expected_response, response.data)
+
+        data = {
+            'scannable_uri_uuid': self.scannable_uri2.uuid,
+            'scan_status': 'invalid'
+        }
+        response = self.scan_queue_worker_client.post('/api/scan_queue/update_status/', data=data)
+        expected_response = {'error': 'invalid scan_status: invalid'}
+        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
+        self.assertEqual(expected_response, response.data)
+
+        data = {
+            'scannable_uri_uuid': 'asdf',
+            'scan_status': 'scanned'
+        }
+        response = self.scan_queue_worker_client.post('/api/scan_queue/update_status/', data=data)
+        expected_response = {'error': 'invalid scannable_uri_uuid: asdf'}
+        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
+        self.assertEqual(expected_response, response.data)
+
+        # Test that staff user can use endpoint
+        data = {
+            'scannable_uri_uuid': 'asdf',
+            'scan_status': 'scanned'
+        }
+        response = self.staff_client.post('/api/scan_queue/update_status/', data=data)
+        expected_response = {'error': 'invalid scannable_uri_uuid: asdf'}
+        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
+        self.assertEqual(expected_response, response.data)
+
+    def test_api_scannable_uri_update_status_update_finished_scannable_uri(self):
+        scannable_uri_uuid = self.scannable_uri3.uuid
+        for scan_status in [
+            ScannableURI.SCAN_INDEXED,
+            ScannableURI.SCAN_FAILED,
+            ScannableURI.SCAN_TIMEOUT,
+            ScannableURI.SCAN_INDEX_FAILED,
+        ]:
+            self.scannable_uri3.scan_status = scan_status
+            self.scannable_uri3.save()
+            data = {
+                'scannable_uri_uuid': scannable_uri_uuid,
+                'scan_status': 'scanned'
+            }
+            response = self.scan_queue_worker_client.post('/api/scan_queue/update_status/', data=data)
+            expected_response = {
+                'error': 'cannot update status for scannable_uri '
+                        f'{self.scannable_uri3.uuid}: scannable_uri has finished '
+                        f'with status "{ScannableURI.SCAN_STATUSES_BY_CODE[scan_status]}"'
+            }
+            self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
+            self.assertEqual(expected_response, response.data)
diff --git a/minecode/tests/test_process_scans.py b/minecode/tests/test_indexing.py
similarity index 51%
rename from minecode/tests/test_process_scans.py
rename to minecode/tests/test_indexing.py
index 442efdcd..24d46b45 100644
--- a/minecode/tests/test_process_scans.py
+++ b/minecode/tests/test_indexing.py
@@ -10,14 +10,9 @@
 import json
 import os
 
-from mock import Mock
-from mock import patch
-
 from matchcode.models import ExactFileIndex
-from minecode.management.commands.process_scans import Command
-from minecode.management.commands.process_scans import get_scan_status
-from minecode.management.commands.process_scans import index_package_files
-from minecode.management.scanning import Scan
+
+from minecode.management import indexing
 from minecode.models import ScannableURI
 from minecode.utils_test import MiningTestCase
 from minecode.utils_test import JsonBasedTesting
@@ -25,7 +20,7 @@
 from packagedb.models import Resource
 
 
-class ProcessScansTest(MiningTestCase, JsonBasedTesting):
+class IndexingTest(MiningTestCase, JsonBasedTesting):
     BASE_DIR = os.path.join(os.path.dirname(__file__), 'testfiles')
 
     def setUp(self):
@@ -37,74 +32,69 @@ def setUp(self):
             version='20040705.181715'
         )
 
-    def test_ProcessScansTest_get_scan_status(self):
-        scan = Scan()
-        scan.status = 'not_started'
-        self.assertEqual(ScannableURI.SCAN_SUBMITTED, get_scan_status(scan))
-        scan.status = 'queued'
-        self.assertEqual(ScannableURI.SCAN_SUBMITTED, get_scan_status(scan))
-        scan.status = 'running'
-        self.assertEqual(ScannableURI.SCAN_IN_PROGRESS, get_scan_status(scan))
-        scan.status = 'success'
-        self.assertEqual(ScannableURI.SCAN_COMPLETED, get_scan_status(scan))
-        scan.status = 'failure'
-        self.assertEqual(ScannableURI.SCAN_FAILED, get_scan_status(scan))
-        scan.status = 'stopped'
-        self.assertEqual(ScannableURI.SCAN_FAILED, get_scan_status(scan))
-        scan.status = 'stale'
-        self.assertEqual(ScannableURI.SCAN_FAILED, get_scan_status(scan))
-        scan.status = 'asdf'
-        self.assertRaises(Exception, get_scan_status, scan)
-
-    def test_ProcessScansTest_index_package_files(self):
+    def test_indexing_index_package_files(self):
         scan_data_loc = self.get_test_loc('scancodeio/get_scan_data.json')
         with open(scan_data_loc, 'rb') as f:
             scan_data = json.loads(f.read())
-        self.assertEqual(0, len(index_package_files(self.package1, scan_data)))
+        self.assertEqual(0, len(indexing.index_package_files(self.package1, scan_data)))
         result = Resource.objects.filter(package=self.package1)
         self.assertEqual(64, len(result))
         results = [r.to_dict() for r in result]
         expected_resources_loc = self.get_test_loc('scancodeio/get_scan_data_expected_resources.json')
         self.check_expected_results(results, expected_resources_loc, regen=False)
 
-    @patch('requests.get')
-    def test_ProcessScansTest_process_scan(self, mock_get):
-        # Set up mock responses
-        mock_scan_info_response = Mock()
-        scan_info_loc = self.get_test_loc('scancodeio/get_scan_info.json')
-        with open(scan_info_loc, 'rb') as f:
-            mock_scan_info_response.json.return_value = json.loads(f.read())
-
-        mock_scan_data_response = Mock()
+    def test_indexing_index_package(self):
         scan_data_loc = self.get_test_loc('scancodeio/get_scan_data.json')
         with open(scan_data_loc, 'rb') as f:
-            mock_scan_data_response.json.return_value = json.loads(f.read())
+            scan_data = json.load(f)
 
-        mock_scan_summary_response = Mock()
         scan_summary_loc = self.get_test_loc('scancodeio/scan_summary_response.json')
         with open(scan_summary_loc, 'rb') as f:
-            mock_scan_summary_response.json.return_value = json.loads(f.read())
+            scan_summary = json.load(f)
 
-        mock_get.side_effect = [mock_scan_info_response, mock_scan_data_response, mock_scan_summary_response]
+        project_extra_data = {
+            'md5': 'md5',
+            'sha1': 'sha1',
+            'sha256': 'sha256',
+            'sha512': 'sha512',
+            'size': 100,
+        }
 
         # Set up ScannableURI
-        scan_uuid = '54dc4afe-70ea-4f1c-9ed3-989efd9a991f'
         scannable_uri = ScannableURI.objects.create(
             uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-            scan_uuid=scan_uuid,
             scan_status=ScannableURI.SCAN_COMPLETED,
             package=self.package1
         )
 
+        self.assertFalse(self.package1.md5)
+        self.assertFalse(self.package1.sha1)
+        self.assertFalse(self.package1.sha256)
+        self.assertFalse(self.package1.sha512)
+        self.assertFalse(self.package1.size)
+        self.assertFalse(self.package1.declared_license_expression)
+        self.assertFalse(self.package1.copyright)
+        self.assertEqual(0, Resource.objects.all().count())
+
         # Run test
-        Command.process_scan(scannable_uri)
+        indexing.index_package(
+            scannable_uri,
+            self.package1,
+            scan_data,
+            scan_summary,
+            project_extra_data,
+        )
 
-        # Make sure that we get license_expression and copyright from the summary
+        # Make sure that Package data is updated
         self.assertEqual('apache-2.0', self.package1.declared_license_expression)
         self.assertEqual('Copyright (c) Apache Software Foundation', self.package1.copyright)
+        self.assertEqual('md5', self.package1.md5)
+        self.assertEqual('sha1', self.package1.sha1)
+        self.assertEqual('sha256', self.package1.sha256)
+        self.assertEqual('sha512', self.package1.sha512)
+        self.assertEqual(100, self.package1.size)
 
         result = Resource.objects.filter(package=self.package1)
         self.assertEqual(64, result.count())
         result = ExactFileIndex.objects.filter(package=self.package1)
         self.assertEqual(45, result.count())
-
diff --git a/minecode/tests/test_request_scans.py b/minecode/tests/test_request_scans.py
deleted file mode 100644
index 4cbf1ad1..00000000
--- a/minecode/tests/test_request_scans.py
+++ /dev/null
@@ -1,73 +0,0 @@
-#
-# Copyright (c) nexB Inc. and others. All rights reserved.
-# purldb is a trademark of nexB Inc.
-# SPDX-License-Identifier: Apache-2.0
-# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
-# See https://github.com/nexB/purldb for support or download.
-# See https://aboutcode.org for more information about nexB OSS projects.
-#
-
-import json
-import os
-
-from mock import Mock
-from mock import patch
-
-from django.db.models import Q
-
-from minecode.management.commands.request_scans import Command
-from minecode.utils_test import MiningTestCase
-from packagedb.models import Package
-from minecode.models import ScannableURI
-
-
-class RequestScansTest(MiningTestCase):
-    BASE_DIR = os.path.join(os.path.dirname(__file__), 'testfiles')
-
-    def setUp(self):
-        self.package1 = Package.objects.create(
-            download_url='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-            namespace='',
-            name='wagon-api',
-            version='20040705.181715'
-        )
-
-    @patch('requests.post')
-    def test_RequestScansTest_request_scan(self, mock_post):
-        # Set up mock responses
-        mock_scan_request_response = Mock()
-        scan_request_loc = self.get_test_loc('scancodeio/scan_request_response.json')
-        with open(scan_request_loc, 'rb') as f:
-            mock_scan_request_response.json.return_value = json.loads(f.read())
-
-        mock_post.side_effect = [mock_scan_request_response]
-
-        # Set up ScannableURI
-        scannable_uri1 = ScannableURI.objects.create(
-            uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-            scan_status=ScannableURI.SCAN_NEW,
-            package=self.package1
-        )
-
-        for scannable_uri in ScannableURI.objects.all():
-            # Run test
-            Command.process_scan(scannable_uri, options={})
-
-        result = ScannableURI.objects.get(uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar')
-        self.assertEqual(ScannableURI.SCAN_SUBMITTED, result.scan_status)
-
-    def test_RequestScansTest_limit_scan_request(self):
-        # Set up Package and ScannableURI
-        package1 = Package.objects.create(download_url='example.com', name='Foo', version='1.23')
-        scannable_uri1 = ScannableURI.objects.create(
-            uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-            scan_status=ScannableURI.SCAN_NEW,
-            package=package1
-        )
-
-        for scannable_uri in ScannableURI.objects.all():
-            # Run test, no API call should be made because `max_scan_requests` is 0
-            Command.process_scan(scannable_uri, options={'max_scan_requests': 0})
-
-        result = ScannableURI.objects.get(uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar')
-        self.assertEqual(ScannableURI.SCAN_NEW, result.scan_status)
diff --git a/minecode/tests/test_scanning.py b/minecode/tests/test_scanning.py
deleted file mode 100644
index 632e4e3b..00000000
--- a/minecode/tests/test_scanning.py
+++ /dev/null
@@ -1,194 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) nexB Inc. and others. All rights reserved.
-# purldb is a trademark of nexB Inc.
-# SPDX-License-Identifier: Apache-2.0
-# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
-# See https://github.com/nexB/purldb for support or download.
-# See https://aboutcode.org for more information about nexB OSS projects.
-#
-
-import json
-import os
-
-import attr
-import mock
-
-from django.test import TestCase as DjangoTestCase
-
-from minecode.management import scanning
-from minecode.utils_test import JsonBasedTesting
-from packagedb.models import Package
-
-
-class ScanCodeIOAPIHelperFunctionTest(JsonBasedTesting, DjangoTestCase):
-    test_data_dir = os.path.join(os.path.dirname(__file__), 'testfiles')
-
-    def setUp(self):
-        self.package1 = Package.objects.create(
-            type='maven',
-            namespace='maven',
-            name='wagon-api',
-            version='20040705.181715',
-            download_url='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-        )
-
-    @mock.patch('requests.get')
-    def testscanning_query_scans(self, mock_get):
-        mock_get.return_value = mock.Mock(ok=True)
-        scan_info_response_loc = self.get_test_loc('scancodeio/scan_request_lookup.json')
-        with open(scan_info_response_loc, 'rb') as f:
-            mock_get.return_value.json.return_value = json.loads(f.read())
-
-        api_url = 'http://127.0.0.1:8001/api/'
-        api_auth_headers = {}
-        uri = 'https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar'
-        response = scanning.query_scans(uri=uri, api_url=api_url, api_auth_headers=api_auth_headers)
-        result = scanning.Scan.from_response(**response)
-
-        expected = scanning.Scan(
-            url='http://127.0.0.1:8001/api/projects/c3b8d1ab-4811-4ced-84af-080997ef1a1a/',
-            uuid='c3b8d1ab-4811-4ced-84af-080997ef1a1a',
-            run_uuid='336e18e3-fd68-4375-9bf2-87090dc5c726',
-            uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-            created_date='2023-05-19T00:45:29.451929Z',
-            task_start_date='2023-05-19T00:45:29.461599Z',
-            task_end_date='2023-05-19T00:45:39.251824Z',
-            task_exitcode=0,
-            status='success',
-            execution_time=9
-        )
-        result = attr.asdict(result)
-        expected = attr.asdict(expected)
-        self.assertEqual(expected, result)
-
-    @mock.patch('requests.post')
-    def testscanning_submit_scan(self, mock_post):
-        test_loc = self.get_test_loc('scancodeio/scan_request_response.json')
-        mock_post.return_value = mock.Mock(ok=True)
-        with open(test_loc, 'rb') as f:
-            mock_post.return_value.json.return_value = json.loads(f.read())
-        api_url = 'http://127.0.0.1:8001/api/'
-        api_auth_headers = {}
-        uri = 'https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar'
-        result = scanning.submit_scan(
-            uri=uri,
-            package=self.package1,
-            api_url=api_url,
-            api_auth_headers=api_auth_headers
-        )
-        expected = scanning.Scan(
-            url='http://127.0.0.1:8001/api/projects/c3b8d1ab-4811-4ced-84af-080997ef1a1a/',
-            uuid='c3b8d1ab-4811-4ced-84af-080997ef1a1a',
-            run_uuid='336e18e3-fd68-4375-9bf2-87090dc5c726',
-            uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-            created_date='2023-05-19T00:45:29.451929Z',
-            task_start_date=None,
-            task_end_date=None,
-            task_exitcode=None,
-            status='not_started',
-            execution_time=None,
-        )
-        expected = attr.asdict(expected)
-        result = attr.asdict(result)
-        self.assertEqual(expected, result)
-
-    @mock.patch('requests.post')
-    @mock.patch('requests.get')
-    def testscanning_submit_scan_uri_exists(self, mock_post, mock_get):
-        self.maxDiff = None
-        mock_post.return_value = mock.Mock(ok=False)
-        scan_request_response_loc = self.get_test_loc('scancodeio/scan_exists_for_uri.json')
-        with open(scan_request_response_loc, 'rb') as f:
-            mock_post.return_value.json.return_value = json.loads(f.read())
-
-        mock_get.return_value = mock.Mock(ok=True)
-        scan_info_response_loc = self.get_test_loc('scancodeio/scan_request_response.json')
-        with open(scan_info_response_loc, 'rb') as f:
-            mock_get.return_value.json.return_value = json.loads(f.read())
-
-        api_url = 'http://127.0.0.1:8001/api/'
-        api_auth_headers = {}
-        uri = 'https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar'
-        result = scanning.submit_scan(
-            uri=uri,
-            package=self.package1,
-            api_url=api_url,
-            api_auth_headers=api_auth_headers
-        )
-
-        expected = scanning.Scan(
-            url='http://127.0.0.1:8001/api/projects/c3b8d1ab-4811-4ced-84af-080997ef1a1a/',
-            uuid='c3b8d1ab-4811-4ced-84af-080997ef1a1a',
-            run_uuid='336e18e3-fd68-4375-9bf2-87090dc5c726',
-            uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-            created_date='2023-05-19T00:45:29.451929Z',
-            task_start_date=None,
-            task_end_date=None,
-            task_exitcode=None,
-            status='not_started',
-            execution_time=None,
-        )
-
-        expected = attr.asdict(expected)
-        result = attr.asdict(result)
-        self.assertEqual(expected, result)
-
-    def testscanning_get_scan_url(self):
-        scan_uuid = '177eb27a-25d2-4ef0-b608-5a84ea9b1ef1'
-        api_url_projects = 'http://127.0.0.1:8001/api/projects/'
-        suffix = 'results'
-        result = scanning.get_scan_url(scan_uuid=scan_uuid, api_url=api_url_projects)
-        expected = 'http://127.0.0.1:8001/api/projects/177eb27a-25d2-4ef0-b608-5a84ea9b1ef1/'
-        self.assertEqual(expected, result)
-        result_with_suffix = scanning.get_scan_url(scan_uuid=scan_uuid, api_url=api_url_projects, suffix=suffix)
-        expected_with_suffix = 'http://127.0.0.1:8001/api/projects/177eb27a-25d2-4ef0-b608-5a84ea9b1ef1/results/'
-        self.assertEqual(expected_with_suffix, result_with_suffix)
-
-    @mock.patch('requests.get')
-    def testscanning_get_scan_info(self, mock_get):
-        test_loc = self.get_test_loc('scancodeio/get_scan_info.json')
-        mock_get.return_value = mock.Mock(ok=True)
-        with open(test_loc, 'rb') as f:
-            mock_get.return_value.json.return_value = json.loads(f.read())
-        scan_uuid = '54dc4afe-70ea-4f1c-9ed3-989efd9a991f'
-        api_url = 'http://127.0.0.1:8001/api/'
-        api_auth_headers = {}
-        result = scanning.get_scan_info(scan_uuid=scan_uuid, api_url=api_url, api_auth_headers=api_auth_headers)
-        expected = scanning.Scan(
-            url='http://127.0.0.1:8001/api/projects/c3b8d1ab-4811-4ced-84af-080997ef1a1a/',
-            uuid='c3b8d1ab-4811-4ced-84af-080997ef1a1a',
-            run_uuid='336e18e3-fd68-4375-9bf2-87090dc5c726',
-            uri='https://repo1.maven.org/maven2/maven/wagon-api/20040705.181715/wagon-api-20040705.181715.jar',
-            created_date='2023-05-19T00:45:29.451929Z',
-            task_start_date='2023-05-19T00:45:29.461599Z',
-            task_end_date='2023-05-19T00:45:39.251824Z',
-            task_exitcode=0,
-            status='success',
-            execution_time=9,
-            md5='57431f2f6d5841eebdb964b04091b8ed',
-            sha1='feff0d7bacd11d37a9c96daed87dc1db163065b1',
-            sha256='05155c2c588ac5922d930eeb1e8a1da896956f4696ae758d110708e9f095baba',
-            sha512='4431f237bcdfee5d2b86b1b3f01c8abaa160d5b7007c63e6281845a3f920d89fdb2e4044f97694ddef91e174d9dd30e5016bbad46eec2d68af200a47e9cedd85',
-            sha1_git='ad18d88bdae8449e7c170f8e7db1bfe336dbb4e0',
-            filename='wagon-api-20040705.181715.jar',
-            size=47069,
-        )
-        expected = attr.asdict(expected)
-        result = attr.asdict(result)
-        self.assertEqual(expected, result)
-
-    @mock.patch('requests.get')
-    def testscanning_get_scan_data(self, mock_get):
-        test_loc = self.get_test_loc('scancodeio/get_scan_data.json')
-        mock_get.return_value = mock.Mock(ok=True)
-        with open(test_loc, 'rb') as f:
-            mock_get.return_value.json.return_value = json.loads(f.read())
-        scan_uuid = '54dc4afe-70ea-4f1c-9ed3-989efd9a991f'
-        api_url = 'http://127.0.0.1:8001/api/'
-        api_auth_headers = {}
-        expected_loc = self.get_test_loc('scancodeio/get_scan_data_expected.json')
-        result = scanning.get_scan_data(scan_uuid=scan_uuid, api_url=api_url, api_auth_headers=api_auth_headers)
-        with open(expected_loc, 'rb') as f:
-            expected = json.loads(f.read())
-        self.assertEqual(expected['files'], result['files'])
diff --git a/minecode/tests/test_utils.py b/minecode/tests/test_utils.py
index 66e0dde9..f242faf8 100644
--- a/minecode/tests/test_utils.py
+++ b/minecode/tests/test_utils.py
@@ -17,6 +17,7 @@
 from minecode.utils_test import JsonBasedTesting
 from minecode.utils import is_int
 from minecode.utils import stringify_null_purl_fields
+from minecode.utils import validate_uuid
 
 
 class UtilsTest(JsonBasedTesting, DjangoTestCase):
@@ -66,3 +67,15 @@ def test_set_purl(self):
     def test_is_int(self):
         self.assertTrue(is_int(0))
         self.assertFalse(is_int('a'))
+
+    def test_validate_uuid(self):
+        invalid_uuid1 = 'invalid'
+        invalid_uuid2 = '123e4567-e89b-12d3-a456-42665544000G'
+        valid_uuid = 'c2cf7ef0-d3be-4011-bda7-8eb4a196eef2'
+
+        for uuid, expected_result in [
+            [invalid_uuid1, False],
+            [invalid_uuid2, False],
+            [valid_uuid, True],
+        ]:
+            self.assertEqual(expected_result, validate_uuid(uuid))
diff --git a/packagedb/tests/test_api.py b/packagedb/tests/test_api.py
index 217d00c2..e6182189 100644
--- a/packagedb/tests/test_api.py
+++ b/packagedb/tests/test_api.py
@@ -12,7 +12,6 @@
 from unittest import mock
 from uuid import uuid4
 
-from django.contrib.postgres.search import SearchVector
 from django.test import TestCase
 from django.urls import reverse
 from django.utils import timezone
@@ -512,25 +511,28 @@ def setUp(self):
         self.scannableuri.scan_uuid = self.scan_uuid
         self.scannableuri.scan_error = 'error'
         self.scannableuri.index_error = 'error'
-        self.scan_request_date = timezone.now()
-        self.scannableuri.scan_request_date = self.scan_request_date
+        self.scan_date = timezone.now()
+        self.scannableuri.scan_date = self.scan_date
 
     def test_reindex_package(self):
-        self.assertEqual(False, self.scannableuri.rescan_uri)
+        self.assertEqual(1, ScannableURI.objects.all().count())
+        response = self.client.get(f'/api/packages/{self.package.uuid}/reindex_package/')
+        self.assertEqual('pkg:maven/sample/Baz@90.12 has been queued for reindexing', response.data['status'])
+        self.assertEqual(2, ScannableURI.objects.all().count())
+        new_scannable_uri = ScannableURI.objects.exclude(pk=self.scannableuri.pk).first()
+        self.assertEqual(self.package, new_scannable_uri.package)
+        self.assertEqual(True, new_scannable_uri.reindex_uri)
+        self.assertEqual(100, new_scannable_uri.priority)
+        self.assertEqual(None, new_scannable_uri.scan_error)
+        self.assertEqual(None, new_scannable_uri.index_error)
+        self.assertEqual(None, new_scannable_uri.scan_date)
+
+        # Ensure previous ScannableURI was not modified
+        self.assertEqual(False, self.scannableuri.reindex_uri)
         self.assertEqual(0, self.scannableuri.priority)
-        self.assertEqual(self.scan_uuid, self.scannableuri.scan_uuid)
         self.assertEqual('error', self.scannableuri.scan_error)
         self.assertEqual('error', self.scannableuri.index_error)
-        self.assertEqual(self.scan_request_date, self.scannableuri.scan_request_date)
-        response = self.client.get(f'/api/packages/{self.package.uuid}/reindex_package/')
-        self.assertEqual('pkg:maven/sample/Baz@90.12 has been queued for reindexing', response.data['status'])
-        self.scannableuri.refresh_from_db()
-        self.assertEqual(True, self.scannableuri.rescan_uri)
-        self.assertEqual(100, self.scannableuri.priority)
-        self.assertEqual(None, self.scannableuri.scan_uuid)
-        self.assertEqual(None, self.scannableuri.scan_error)
-        self.assertEqual(None, self.scannableuri.index_error)
-        self.assertEqual(None, self.scannableuri.scan_request_date)
+        self.assertEqual(self.scan_date, self.scannableuri.scan_date)
 
 
 class PackageApiPurlFilterTestCase(JsonBasedTesting, TestCase):
@@ -724,7 +726,7 @@ class CollectApiTestCase(JsonBasedTesting, TestCase):
     test_data_dir = os.path.join(os.path.dirname(__file__), 'testfiles')
 
     def setUp(self):
-        package_download_url = 'http://anotherexample.com'
+        self.package_download_url = 'http://anotherexample.com'
         self.package_data = {
             'type': 'maven',
             'namespace': 'sample',
@@ -732,7 +734,7 @@ def setUp(self):
             'version': '90.12',
             'qualifiers': '',
             'subpath': '',
-            'download_url': package_download_url,
+            'download_url': self.package_download_url,
             'filename': 'Baz.zip',
             'sha1': 'testsha1-3',
             'md5': 'testmd5-3',
@@ -742,9 +744,9 @@ def setUp(self):
         self.package.refresh_from_db()
         self.scannableuri = ScannableURI.objects.create(
             package=self.package,
-            uri=package_download_url,
+            uri=self.package_download_url,
         )
-        self.scannableuri.scan_status = ScannableURI.SCAN_INDEXED
+        self.scannableuri.scan_status = ScannableURI.SCAN_INDEX_FAILED
         self.scan_uuid = uuid4()
         self.scannableuri.scan_uuid = self.scan_uuid
         self.scannableuri.scan_error = 'error'
@@ -752,7 +754,7 @@ def setUp(self):
         self.scan_request_date = timezone.now()
         self.scannableuri.scan_request_date = self.scan_request_date
 
-        package_download_url2 = 'http://somethingelse.org'
+        self.package_download_url2 = 'http://somethingelse.org'
         self.package_data2 = {
             'type': 'npm',
             'namespace': 'example',
@@ -760,7 +762,7 @@ def setUp(self):
             'version': '56.78',
             'qualifiers': '',
             'subpath': '',
-            'download_url': package_download_url2,
+            'download_url': self.package_download_url2,
             'filename': 'Bar.zip',
             'sha1': 'testsha1-2',
             'md5': 'testmd5-2',
@@ -770,16 +772,16 @@ def setUp(self):
         self.package2.refresh_from_db()
         self.scannableuri2 = ScannableURI.objects.create(
             package=self.package2,
-            uri=package_download_url2,
+            uri=self.package_download_url2,
         )
-        self.scannableuri2.scan_status = ScannableURI.SCAN_INDEXED
+        self.scannableuri2.scan_status = ScannableURI.SCAN_INDEX_FAILED
         self.scan_uuid2 = uuid4()
         self.scannableuri2.scan_uuid = self.scan_uuid2
         self.scannableuri2.scan_error = 'error'
         self.scannableuri2.index_error = 'error'
         self.scan_request_date2 = timezone.now()
         self.scannableuri2.scan_request_date = self.scan_request_date2
-    
+
     def test_package_live(self):
         purl_str = 'pkg:maven/org.apache.twill/twill-core@0.12.0'
         download_url = 'https://repo1.maven.org/maven2/org/apache/twill/twill-core/0.12.0/twill-core-0.12.0.jar'
@@ -805,7 +807,7 @@ def test_package_live(self):
         ]
 
         self.check_expected_results(result, expected, fields_to_remove=fields_to_remove, regen=False)
-    
+
     def test_package_api_index_packages_endpoint(self):
         priority_resource_uris_count = PriorityResourceURI.objects.all().count()
         self.assertEqual(0, priority_resource_uris_count)
@@ -980,21 +982,25 @@ def test_package_api_index_packages_endpoint_all_version_index(self, mock_get_al
         self.assertEqual([], response.data["unsupported_packages"])
         priority_resource_uris_count = PriorityResourceURI.objects.all().count()
         self.assertEqual(13, priority_resource_uris_count)
-    
+
     def test_reindex_packages_bulk(self):
-        self.assertEqual(False, self.scannableuri.rescan_uri)
+        self.assertEqual(2, ScannableURI.objects.all().count())
+
+        self.assertEqual(False, self.scannableuri.reindex_uri)
         self.assertEqual(0, self.scannableuri.priority)
         self.assertEqual(self.scan_uuid, self.scannableuri.scan_uuid)
         self.assertEqual('error', self.scannableuri.scan_error)
         self.assertEqual('error', self.scannableuri.index_error)
         self.assertEqual(self.scan_request_date, self.scannableuri.scan_request_date)
+        self.assertEqual(ScannableURI.SCAN_INDEX_FAILED, self.scannableuri.scan_status)
 
-        self.assertEqual(False, self.scannableuri2.rescan_uri)
+        self.assertEqual(False, self.scannableuri2.reindex_uri)
         self.assertEqual(0, self.scannableuri2.priority)
         self.assertEqual(self.scan_uuid2, self.scannableuri2.scan_uuid)
         self.assertEqual('error', self.scannableuri2.scan_error)
         self.assertEqual('error', self.scannableuri2.index_error)
         self.assertEqual(self.scan_request_date2, self.scannableuri2.scan_request_date)
+        self.assertEqual(ScannableURI.SCAN_INDEX_FAILED, self.scannableuri2.scan_status)
 
         packages = [
             # Existing package
@@ -1015,7 +1021,7 @@ def test_reindex_packages_bulk(self):
             'pkg:maven/sample/Baz@90.12',
             'pkg:npm/example/bar@56.78',
         ]
-        
+
         unsupported_purls = [
             'pkg:pypi/does/not-exist@1',
         ]
@@ -1033,23 +1039,18 @@ def test_reindex_packages_bulk(self):
         self.assertEqual(0, response.data["unqueued_packages_count"])
         self.assertEqual([], response.data["unqueued_packages"])
 
+        self.assertEqual(4, ScannableURI.objects.all().count())
+        new_scannable_uris = ScannableURI.objects.exclude(pk__in=[self.scannableuri.pk, self.scannableuri2.pk])
+        self.assertEqual(2, new_scannable_uris.count())
 
-        self.scannableuri.refresh_from_db()
-        self.assertEqual(True, self.scannableuri.rescan_uri)
-        self.assertEqual(100, self.scannableuri.priority)
-        self.assertEqual(None, self.scannableuri.scan_uuid)
-        self.assertEqual(None, self.scannableuri.scan_error)
-        self.assertEqual(None, self.scannableuri.index_error)
-        self.assertEqual(None, self.scannableuri.scan_request_date)
+        for scannable_uri in new_scannable_uris:
+            self.assertEqual(True, scannable_uri.reindex_uri)
+            self.assertEqual(100, scannable_uri.priority)
+            self.assertEqual(ScannableURI.SCAN_NEW, scannable_uri.scan_status)
+            self.assertEqual(None, scannable_uri.scan_error)
+            self.assertEqual(None, scannable_uri.index_error)
+            self.assertEqual(None, scannable_uri.scan_date)
 
-        self.scannableuri2.refresh_from_db()
-        self.assertEqual(True, self.scannableuri2.rescan_uri)
-        self.assertEqual(100, self.scannableuri.priority)
-        self.assertEqual(None, self.scannableuri2.scan_uuid)
-        self.assertEqual(None, self.scannableuri2.scan_error)
-        self.assertEqual(None, self.scannableuri2.index_error)
-        self.assertEqual(None, self.scannableuri2.scan_request_date)
-    
 
 class ResourceApiTestCase(TestCase):
 
@@ -1126,7 +1127,7 @@ def setUp(self):
         }
         self.package = Package.objects.create(**self.package_data)
         self.package.refresh_from_db()
-    
+
     def test_api_purl_validation(self):
         data1 = {
             "purl": "pkg:npm/foobar@1.1.0",
@@ -1151,7 +1152,7 @@ def test_api_purl_validation(self):
         self.assertEqual(
             "The provided PackageURL is not valid.", response2.data["message"]
         )
-    
+
     def test_api_purl_validation_unsupported_package_type(self):
         data1 = {
             "purl": "pkg:random/foobar@1.1.0",
@@ -1169,7 +1170,7 @@ def test_api_purl_validation_unsupported_package_type(self):
     def test_api_purl_validation_empty_request(self):
         data1 = {}
         response1 = self.client.get(f"/api/validate/", data=data1)
-        
+
         expected = {
             "errors": {
                 "purl": [
diff --git a/packagedb/tests/testfiles/api/resource-filter_by_checksums-expected.json b/packagedb/tests/testfiles/api/resource-filter_by_checksums-expected.json
index 325beb1c..4582b369 100644
--- a/packagedb/tests/testfiles/api/resource-filter_by_checksums-expected.json
+++ b/packagedb/tests/testfiles/api/resource-filter_by_checksums-expected.json
@@ -11,9 +11,9 @@
     "sha256":"testsha2561",
     "sha512":"testsha5121",
     "git_sha1":"testgit_sha11",
-    "mime_type":"",
-    "file_type":"",
-    "programming_language":"",
+    "mime_type":null,
+    "file_type":null,
+    "programming_language":null,
     "is_binary":false,
     "is_text":false,
     "is_archive":false,
@@ -44,9 +44,9 @@
     "sha256":"testsha2562",
     "sha512":"testsha5122",
     "git_sha1":"testgit_sha12",
-    "mime_type":"",
-    "file_type":"",
-    "programming_language":"",
+    "mime_type":null,
+    "file_type":null,
+    "programming_language":null,
     "is_binary":false,
     "is_text":false,
     "is_archive":false,