Repository: weecology/retriever

Issue:
Search for datasets by license
We currently store license data in the `licenses` field for some datasets. It would be useful to be able to find datasets matching a particular license or set of licenses, e.g.,

CLI:
```
$ retriever ls -l MIT GPL
```

Python interface:
```
>>> retriever.dataset_names(licenses = ["MIT", "GPL"])
```

This should return a list of datasets with one of the specified licenses.

Similar CLI functionality exists at https://github.com/weecology/retriever/blob/master/retriever/__main__.py#L140 for seaching based on keywords.
Search for datasets by license
We currently store license data in the `licenses` field for some datasets. It would be useful to be able to find datasets matching a particular license or set of licenses, e.g.,

CLI:
```
$ retriever ls -l MIT GPL
```

Python interface:
```
>>> retriever.dataset_names(licenses = ["MIT", "GPL"])
```

This should return a list of datasets with one of the specified licenses.

Similar CLI functionality exists at https://github.com/weecology/retriever/blob/master/retriever/__main__.py#L140 for seaching based on keywords.


Discussion:
Hi @ethanwhite 
I will to work on this issue. How do I get started?
Hi @adhaamehab the best way to start is by looking at the developer docs on how to set up the environment. Let me know if you need any help when setting up the environment 
Hi @henrykironde  @ethanwhite 
I'm a bit confused with issue #969 
```
retriever ls -l MIT
``` 
 will print

```Available datasets : 1

1. Masses of Mammals (Smith et al. 2003)
mammal-masses
['mammals', 'literature-compilation', 'size']
```
so what should this command output ? how should I differentiate between keyword and license? 
should I first check if the input -or set of inputs- is a licenses ?
We are using `argparse` in `retriever/retriever/lib/get_opts.py` to parse arguments.
Right now, for this specific command `retriever ls -l MIT`, Retriever searches for dataset that have keywords that are like MIT.
_We need to make it better actually, because MIT is not so close to the results provided._

However, If we add licenses, then we should adjust the code to look for datasets(scripts) that also contain that given license.

There are two designs we could follow

1. Using `retriever ls -l `  like in retriever ls -l MIT  and we just add some logic to look for licenses
2. Using a new option for only licenses. e.g `retriever ls --license MIT` or `retriever ls -c MIT`

I think the first option would be good for now. We can change it later, if we want something specific. 
Let me know if that sounds clear 


Hi @ethanwhite 
I will to work on this issue. How do I get started?
Hi @adhaamehab the best way to start is by looking at the developer docs on how to set up the environment. Let me know if you need any help when setting up the environment 
Hi @henrykironde  @ethanwhite 
I'm a bit confused with issue #969 
```
retriever ls -l MIT
``` 
 will print

```Available datasets : 1

1. Masses of Mammals (Smith et al. 2003)
mammal-masses
['mammals', 'literature-compilation', 'size']
```
so what should this command output ? how should I differentiate between keyword and license? 
should I first check if the input -or set of inputs- is a licenses ?
We are using `argparse` in `retriever/retriever/lib/get_opts.py` to parse arguments.
Right now, for this specific command `retriever ls -l MIT`, Retriever searches for dataset that have keywords that are like MIT.
_We need to make it better actually, because MIT is not so close to the results provided._

However, If we add licenses, then we should adjust the code to look for datasets(scripts) that also contain that given license.

There are two designs we could follow

1. Using `retriever ls -l `  like in retriever ls -l MIT  and we just add some logic to look for licenses
2. Using a new option for only licenses. e.g `retriever ls --license MIT` or `retriever ls -c MIT`

I think the first option would be good for now. We can change it later, if we want something specific. 
Let me know if that sounds clear 



Resolving patch:
diff --git a/retriever/__main__.py b/retriever/__main__.py
--- a/retriever/__main__.py
+++ b/retriever/__main__.py
@@ -132,21 +132,30 @@ def main():
                 check_for_updates(False)
                 print("\n\nScripts downloaded.\n")
 
-            if args.l is None:
-                all_scripts = datasets()
+            if args.l is None and args.k is None:
+                all_scripts = dataset_names()
                 print("Available datasets : {}\n".format(len(all_scripts)))
                 from retriever import lscolumns
-                lscolumns.printls(dataset_names())
+                lscolumns.printls(all_scripts)
             else:
-                all_scripts = datasets(args.l[0])
-                print("Available datasets : {}\n".format(len(all_scripts)))
-                count = 1
-                for script in all_scripts:
-                    print("{}. {}".format(count, script.title))
-                    print(script.name)
-                    print(script.keywords)
-                    print()
-                    count += 1
+                param_licenses = args.l if args.l else None
+                keywords = args.k if args.k else None
+
+                # search
+                searched_scripts = datasets(keywords, param_licenses)
+                if not searched_scripts:
+                    print("No available datasets found")
+                else:
+                    print("Available datasets : {}\n".format(len(searched_scripts)))
+                    count = 1
+                    for script in searched_scripts:
+                        print("{}. {}\n{}\n{}\n{}\n".format(
+                                count, script.title,
+                                script.name,
+                                script.keywords,
+                                str(script.licenses[0]['name'])
+                        ))
+                        count += 1
             return
 
         engine = choose_engine(args.__dict__)
diff --git a/retriever/lib/datasets.py b/retriever/lib/datasets.py
--- a/retriever/lib/datasets.py
+++ b/retriever/lib/datasets.py
@@ -1,26 +1,33 @@
 from retriever.lib.scripts import SCRIPT_LIST, get_script
 
 
-def datasets(arg_keyword=None):
-    """Return list of all available datasets."""
+def datasets(keywords=None, licenses=None):
+    """Search all datasets by keywords and licenses."""
     script_list = SCRIPT_LIST()
 
-    all_scripts = []
+    if not keywords and not licenses:
+        return sorted(script_list, key=lambda s: s.name.lower())
 
+    result_scripts = set()
+    if licenses:
+        licenses = [l.lower() for l in licenses]
     for script in script_list:
         if script.name:
-            if arg_keyword:
-                keywords = script.title + ',' + script.name
+            if licenses:
+                script_license = script.licenses[0]['name']
+                if script_license and script_license.lower() in licenses:
+                    result_scripts.add(script)
+                    continue
+            if keywords:
+                script_keywords = script.title + ' ' + script.name
                 if script.keywords:
-                    keywords = keywords + ',' + ','.join(script.keywords)
-                if keywords.lower().find(arg_keyword.lower()) != -1:
-                    all_scripts.append(script)
-            else:
-                all_scripts.append(script)
-
-    all_scripts = sorted(all_scripts, key=lambda s: s.name.lower())
-
-    return all_scripts
+                    script_keywords = script_keywords + ' ' + '-'.join(script.keywords)
+                script_keywords = script_keywords.lower()
+                for k in keywords:
+                    if script_keywords.find(k.lower()) != -1:
+                        result_scripts.add(script)
+                        break
+    return sorted(list(result_scripts), key=lambda s: s.name.lower())
 
 
 def dataset_names():
@@ -37,3 +44,9 @@ def dataset_names():
 def license(dataset):
     """Get the license for a dataset."""
     return get_script(dataset).licenses[0]['name']
+
+
+def dataset_licenses():
+    """Return set with all available licenses."""
+    return set([
+        str(script.licenses[0]['name']).lower() for script in SCRIPT_LIST()])
diff --git a/retriever/lib/get_opts.py b/retriever/lib/get_opts.py
--- a/retriever/lib/get_opts.py
+++ b/retriever/lib/get_opts.py
@@ -13,10 +13,16 @@
 json_list = [module.name for module in module_list
              if os.path.isfile('.'.join(module._file.split('.')[:-1]) + '.json')]
 
-keywords_list = set()
+# set of all possible licenses for ls completion
+licenses_options = set()
+keywords_options = set()
 for module in module_list:
     if hasattr(module, "keywords"):
-        keywords_list = keywords_list | set(module.keywords)
+        keywords_options = keywords_options | set(module.keywords)
+    if hasattr(module, "licenses"):
+        if module.licenses[0]['name']:
+            licenses_options = licenses_options | set([module.licenses[0]['name']])
+
 
 parser = argparse.ArgumentParser(prog="retriever")
 parser.add_argument('-v', '--version', action='version', version=VERSION)
@@ -59,9 +65,12 @@
 install_parser.add_argument('--debug', help='run in debug mode', action='store_true')
 install_parser.add_argument('--not-cached', help='overwrites local cache of raw data', action='store_true')
 download_parser.add_argument('dataset', help='dataset name').completer = ChoicesCompleter(script_list)
-ls_parser.add_argument('-l', help='verbose list of datasets containing following keywords '
-                                  '(lists all when no keywords are specified)',
-                       nargs=1).completer = ChoicesCompleter(list(keywords_list))
+ls_parser.add_argument('-l', help='search datasets with specific license(s)',
+                       nargs='+').completer = ChoicesCompleter(list(licenses_options))
+ls_parser.add_argument('-k', help='search datasets with keyword(s)',
+                       nargs='+').completer = ChoicesCompleter(list(keywords_options))
+
+
 delete_json_parser.add_argument('dataset', help='dataset name', choices=json_list)
 # retriever Install {Engine} ..
 # retriever download [options]


Test patch: