Repository: opendatacube/datacube-core

Issue:
Allow indexing without provenance tracking
### Rationale
For testing and debugging, it would be helpful if `datacube` allowed indexing datasets
with lineage information stripped off. This would help avoid filling test datacubes to hold,
for example, simple statistics on a product with irrelevant lineage data tracing back to
acquisition.

### Expected behaviour
Something along the lines of
`$ datacube dataset add --no-lineage DATASET`
should index dataset with `lineage.source_datasets` set to `{}`.

### Environment information
* Which ``datacube --version`` are you using?
Open Data Cube core, version 1.5.1+516.g7046071





Discussion:
We already have a command line option of `--sources-policy`. How about adding another policy named `drop`. eg:
```bash
$ datacube dataset add --sources-policy drop <datasets>
```
But what does `--sources-policy=X` actually do? I remember expecting something else from it than what it actually does.
It already has "skip" which is very close to "drop", but what it skips is the dataset being added, not the sources dataset.
Also there is both "verify" and "ensure", but which one ensures more?
There's a docstring the explains it, but perhaps it belongs in one of the guides too?

`ensure`: ensure that it exists in the index (adding it if needed)

`verify` (default): ensure that it exists in the index (adding it if needed), and throw an error if any metadata differs from what we already have.

`skip`: Confusingly named: it skips adding the sources, but will still link them to this dataset.
- From a user perspective, `skip` really means "throw an error if we don't already have all the sources".
- Which is slightly more useful than it sounds: most of our processing is on datasets that are already in the cube, so if it's quietly adding new sources we'd want to know about it.
    - Example previous bug: an old scene processor that inappropriately scanned the filesystem for datasets to process was picking up datasets that were partially-written/corrupt.
- I think it was actually added before `ensure` existed, as brute-force way to "skip" the metadata verification. `ensure` is a more forgiving alternative.

Resolving patch:
diff --git a/datacube/scripts/dataset.py b/datacube/scripts/dataset.py
--- a/datacube/scripts/dataset.py
+++ b/datacube/scripts/dataset.py
@@ -91,7 +91,7 @@ def create_dataset(dataset_doc, uri, rules, skip_lineage=False):
     """
     dataset_type = find_matching_product(rules, dataset_doc)
     if skip_lineage:
-        sources = None
+        sources = {}
         dataset_doc = without_lineage_sources(dataset_doc, dataset_type)
     else:
         sources = {cls: create_dataset(source_doc, None, rules)
@@ -212,10 +212,32 @@ def parse_match_rules_options(index, dtype, auto_match):
 'ensure' - add source dataset if it doesn't exist
 'skip' - dont add the derived dataset if source dataset doesn't exist""")
 @click.option('--dry-run', help='Check if everything is ok', is_flag=True, default=False)
+@click.option('--ignore-lineage',
+              help="Don't add lineage data to the database",
+              is_flag=True, default=False)
+@click.option('--confirm-ignore-lineage',
+              help="Don't add lineage data to the database, without confirmation",
+              is_flag=True, default=False)
 @click.argument('dataset-paths',
                 type=click.Path(exists=True, readable=True, writable=False), nargs=-1)
 @ui.pass_index()
-def index_cmd(index, product_names, auto_match, sources_policy, dry_run, dataset_paths):
+def index_cmd(index, product_names, auto_match, sources_policy, dry_run,
+              ignore_lineage,
+              confirm_ignore_lineage,
+              dataset_paths):
+
+    if confirm_ignore_lineage is False and ignore_lineage is True:
+        if sys.stdin.isatty():
+            confirmed = click.confirm("Requested to skip lineage information, Are you sure?", default=False)
+            if not confirmed:
+                click.echo('OK aborting', err=True)
+                sys.exit(1)
+        else:
+            click.echo("Use --confirm-ignore-lineage from non-interactive scripts. Aborting.")
+            sys.exit(1)
+
+        confirm_ignore_lineage = True
+
     rules = parse_match_rules_options(index, product_names, auto_match)
     if rules is None:
         return
@@ -223,13 +245,15 @@ def index_cmd(index, product_names, auto_match, sources_policy, dry_run, dataset
     # If outputting directly to terminal, show a progress bar.
     if sys.stdout.isatty():
         with click.progressbar(dataset_paths, label='Indexing datasets') as dataset_path_iter:
-            index_dataset_paths(sources_policy, dry_run, index, rules, dataset_path_iter)
+            index_dataset_paths(sources_policy, dry_run, index, rules, dataset_path_iter,
+                                skip_lineage=confirm_ignore_lineage)
     else:
-        index_dataset_paths(sources_policy, dry_run, index, rules, dataset_paths)
+        index_dataset_paths(sources_policy, dry_run, index, rules, dataset_paths,
+                            skip_lineage=confirm_ignore_lineage)
 
 
-def index_dataset_paths(sources_policy, dry_run, index, rules, dataset_paths):
-    for dataset in load_datasets(dataset_paths, rules):
+def index_dataset_paths(sources_policy, dry_run, index, rules, dataset_paths, skip_lineage=False):
+    for dataset in load_datasets(dataset_paths, rules, skip_lineage=skip_lineage):
         _LOG.info('Matched %s', dataset)
         if not dry_run:
             try:


Test patch:
diff --git a/integration_tests/test_end_to_end.py b/integration_tests/test_end_to_end.py
--- a/integration_tests/test_end_to_end.py
+++ b/integration_tests/test_end_to_end.py
@@ -85,6 +85,11 @@ def test_end_to_end(clirunner, index, testdata_dir, ingest_configs):
     clirunner(['-v', 'dataset', 'add',
                str(lbg_nbar), str(lbg_pq)])
 
+    #  - this will be no-op but with ignore lineage
+    clirunner(['-v', 'dataset', 'add',
+               '--confirm-ignore-lineage',
+               str(lbg_nbar), str(lbg_pq)])
+
     # Test no-op update
     for policy in ['archive', 'forget', 'keep']:
         clirunner(['-v', 'dataset', 'update',