Repository: iBridges-for-iRODS/iBridges (language: python)

Issue:
Add search functionality to the CLI
Probably after #229 is done.


Discussion / hints:

Fix patch:
diff --git a/docs/source/api/user_reference.rst b/docs/source/api/user_reference.rst
index ceed06d4..7408c401 100644
--- a/docs/source/api/user_reference.rst
+++ b/docs/source/api/user_reference.rst
@@ -55,6 +55,7 @@ Searching for data
     :toctree: generated/
 
     search_data
+    MetaSearch
 
 
 Tickets
diff --git a/docs/source/cli.rst b/docs/source/cli.rst
index 39668b9d..451b217a 100644
--- a/docs/source/cli.rst
+++ b/docs/source/cli.rst
@@ -7,6 +7,15 @@ that are available in the iBridges API library. It is mainly there for users tha
 and still want to download or upload their data using the interface, or if you need a simple iBridges operation
 in your shell script without having to create a new python script.
 
+.. note::
+
+    All of the subcommands mentioned below have a :code:`--help` flag to explain
+    how the subcommand works:
+
+    .. code:: shell
+
+        ibridges <subcommand> --help
+
 .. note::
 
     There are no CLI commands to add/change metadata, instead use the iBridges API for this.
@@ -162,8 +171,106 @@ that are missing or have a different checksum (content).
     ibridges sync some_local_directory "irods:~/remote_collection"
 
 
-.. note::
+.. warning::
 
     The order of the directory/collection that you supply to :code:`ibridges sync` matters. The first argument is the `source`
     directory/collection, while the second argument is the `destination` directory/collection. Transfers will only happen
     from `source` to `destination`, so extra or updated files in the `destination` directory will not be transferred.
+
+
+Searching for data
+------------------
+
+It can be helpful to search for data if the exact location is not known. This is done
+using the :code:`search` subcommand. There are four different criteria types for searching:
+path pattern, checksum, metadata and item type. By default, the search is conducted in the home directory,
+but this can be modified by supplying a remote path:
+
+.. code:: shell
+
+    ibridges search irods:some_collection # Search criteria after this
+
+.. note::
+
+    The different matching criteria can be combined. If they are combined all of the
+    criteria must be true for the item to show up in the list.
+
+Searching by path pattern
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Searching by path pattern can search for full or partial names of objects and collections.
+For example, to find all :code:`.txt` data objects:
+
+.. code:: shell
+
+    ibridges search --path-pattern "%.txt"
+    
+Find all :code:`.txt` data objects in a collection :code:`demo`
+
+.. code:: shell
+
+    ibridges search --path-pattern "%/demo/%.txt"
+
+
+Searching by checksum
+^^^^^^^^^^^^^^^^^^^^^
+
+Searching for checksum can be useful to find duplicates of data objects:
+
+.. code:: shell
+
+    ibridges search --checksum "5dfasd%"
+
+
+Searching by metadata
+^^^^^^^^^^^^^^^^^^^^^
+
+Metadata can make data more findable. For example, to find all data objects and
+collections that have a metadata entry "key":
+
+
+.. code:: shell
+
+    ibridges search --metadata "key"
+
+The same can be done for finding metadata with a certain key/value pair:
+
+.. code:: shell
+
+    ibridges search --metadata "key" "value"
+
+Wildcards (:code:`%`) can be particularly useful. For example if we want to
+find items with units "kg", we can do:
+
+.. code:: shell
+
+    ibridges search --metadata "%" "%" "kg"
+
+The metadata criterium can be used multiple times:
+
+.. code:: shell
+
+    ibridges search --metadata "key" "value" --metadata "key2" "value2"
+
+Note that in the above example, it is not sufficient for the item to contain
+the keys "key" and "key2", and the values "value" and "value2": the entries
+must have the key/value pairs as indicated in the command.
+
+
+Searching by item type
+^^^^^^^^^^^^^^^^^^^^^^
+
+By default, the search will return both data objects and collections.
+Sometimes it might be useful to only search for collections or data objects.
+In this case, you can use the :code:`--item_type` flag:
+
+
+.. code:: shell
+
+    ibridges search --metadata "key" --item_type collection
+
+or
+
+.. code:: shell
+
+    ibridges search --metadata "key" --item_type data_object
diff --git a/docs/source/irods_search.rst b/docs/source/irods_search.rst
index 717258e3..fa363e78 100644
--- a/docs/source/irods_search.rst
+++ b/docs/source/irods_search.rst
@@ -1,49 +1,72 @@
 iRODS Search
 ============
 
-`iBridges` offers an easy way to search for data. You can pass a combination of path, metadata keys and values and checksum. The output will be a list of dictionaries, one dictionary for each found item, which contain information where to find the item on the iRODS server.
+`iBridges` offers an easy way to search for data. You can pass a combination of path, metadata,
+item type and checksum. The output will be a list of :class:`ibridges.path.IrodsPath`, which contain information where to find the item on the iRODS server.
 
+.. note::
 
-Search data by Path
--------------------
+	The collection to search within is by default your home collection. You can change this
+	by supplying the :code:`path` argument. Note that this :code:`path` itself will not show
+	up in the results.
 
-In the example below we search for a data object by its path.
-The path can be an :code:`IrodsPath` or a string:
+Search data by path pattern
+---------------------------
+
+In the example below we search for a data object by its path pattern.
+The path is a string:
  	
 .. code-block:: python
 		
         from ibridges import search_data
-	    search_data(session, path=IrodsPath(session, "dataobj_name"))
-	
-The result is a list of dictionaries.
+	    search_data(session, path="/", path_pattern="dataobj_name")
 	
-.. code-block:: python
+The result is a list of iRODS paths that indicate the locations of the found collections and data objects.
 	
-		[{'COLL_NAME': '/you/irods/home',
-  		'DATA_NAME': 'dataobj_name',
-  		'D_DATA_CHECKSUM': 'sha2:XGiECYZOtUfP9lnCGyZaBBkBGLaJJw1p6eoc0GxLeKU='}]
-  		
-For collections the dictionary only contains the entry `COLL_NAME`.
-  	
 To find all subcollections and dataobjects in a collection use the `%` as wildcard:
   	
 .. code-block:: python
   	
-  		search_data(session, path=IrodsPath(session, "subcoll/%"))
+  		search_data(session, path_pattern="subcoll/%")
   	
 
 Search data by metadata
 -----------------------
 
 We can also use the metadata we generated above to search for data.
-We need to create a python dictionary which contains the metadata keys ad their values. The values are again optional.
+
+
+To search by metadata we need to create a :class:`ibridges.search.MetaSearch` for each key, value, units triple:
+
+.. code-block:: python
+
+	MetaSearch(key="my_key", value="my_value", units="my_units")
+
+The above statement means: find all data objects and collections which is annotated with a key "my_key" where the value is "my_value" and the units is "my_units".
+
+You can also omit any of the three items which will be interpreted as "anything". E.g.
+
+.. code-block:: python
+
+	MetaSearch(value="my_value")
+
+means: find all data which is labeled with any key where the value is "my_value" and the units can also be anything. Here again you can also use wild cards.
+
+A query with metadata will look like:
+
 
 .. code-block:: python
 
-	key_vals = {'key': 'value'}
-	search_data(session, key_vals = key_vals)
-	key_vals = {'key': ''}
-	search_data(session, key_vals = key_vals)
+	# Search for items with an entry that has key=="key" and any value or units.
+	search_data(session, metadata=MetaSearch(key="key"))
+
+	# Search for items with an entry that has key=="key" and value=="value"
+	search_data(session, metadata=MetaSearch(key="key", value="value"))
+
+	# Different from above! Search for items with one metadata entry that has key=="key"
+	# and one metadata entry that has value=="value", but they do not have to be
+	# for the same entry as in the above.
+	search_data(session, metadata=[MetaSearch(key="key"), MetaSearch(value="value")])
 	
 Use the `%` as a wild card again to match any combination of characters.
 	
@@ -51,9 +74,20 @@ Use the `%` as a wild card again to match any combination of characters.
 Search data by checksum
 -----------------------
 
-The search by checksum is handy to find duplicate data. in *iBridges* we always create checksums when data is uploaded. The checksum is unique for the file we uploaded and can be used to identify the files. E.g. when you uploaded the same file twice, once as `file.txt` and once as `file1.txt` you can find the two duplicates by their checksum:
+The search by checksum is handy to find duplicate data. In *iBridges* we always create checksums when data is uploaded. The checksum is unique for the file we uploaded and can be used to identify the files. E.g. when you uploaded the same file twice, once as `file.txt` and once as `file1.txt` you can find the two duplicates by their checksum:
 
 .. code-block:: python
 
 	obj = IrodsPath(session, "~", "dataobj_name").dataobject
 	search_data(session, checksum = obj.checksum)
+
+Search data by item type
+------------------------
+
+Sometimes you might want to only look for data objects or collections. In
+this case you can select for that:
+
+.. code-block:: python
+
+	search_data(session, path_pattern="sta%", item_type="data_object")
+	search_data(session, path_pattern="sta%", item_type="collection")
diff --git a/ibridges/__main__.py b/ibridges/__main__.py
index 908fdcfd..051cd718 100644
--- a/ibridges/__main__.py
+++ b/ibridges/__main__.py
@@ -5,12 +5,14 @@
 import argparse
 import json
 import sys
+from argparse import RawTextHelpFormatter
 from pathlib import Path
 from typing import Optional, Union
 
 from ibridges.data_operations import download, sync, upload
 from ibridges.interactive import DEFAULT_IENV_PATH, DEFAULT_IRODSA_PATH, interactive_auth
 from ibridges.path import IrodsPath
+from ibridges.search import search_data
 from ibridges.session import Session
 from ibridges.util import (
     find_environment_provider,
@@ -107,6 +109,8 @@ def main() -> None:
         ibridges_tree()
     elif subcommand == "setup":
         ibridges_setup()
+    elif subcommand == "search":
+        ibridges_search()
     else:
         print(f"Invalid subcommand ({subcommand}). For help see ibridges --help")
         sys.exit(1)
@@ -672,3 +676,70 @@ def ibridges_tree():
         if args.depth is not None:
             print_str += " (possibly more at higher depths)"
         print(print_str)
+
+
+def ibridges_search():
+    """Search for collections and objects using constraints."""
+    epilog = """Examples:
+
+ibridges search --path-pattern "%.txt"
+ibridges search --checksum "sha2:5dfasd%"
+ibridges search --metadata "key" "value" "units"
+ibridges search --metadata "key" --metadata "key2" "value2"
+ibridges search irods:some_collection --item_type data_object
+ibridges search irods:some_collection --item_type collection
+"""
+    parser = argparse.ArgumentParser(
+        prog="ibridges search",
+        description="Search for dataobjects and collections.",
+        epilog=epilog,
+        formatter_class=RawTextHelpFormatter,
+    )
+    parser.add_argument(
+        "remote_path",
+        help="Remote path to search inn. The path itself will not be matched.",
+        type=str,
+        default=None,
+        nargs="?"
+    )
+    parser.add_argument(
+        "--path-pattern",
+        default=None,
+        type=str,
+        help=("Pattern of the path constraint. For example, use '%%.txt' to find all data objects"
+              " and collections that end with .txt. You can also use the name of the item here "
+              "to find all items with that name.")
+    )
+    parser.add_argument(
+        "--checksum",
+        default=None,
+        type=str,
+        help="Checksum of the data objects to be found."
+    )
+    parser.add_argument(
+        "--metadata",
+        nargs="+",
+        action="append",
+        help="Constrain the results using metadata, see examples. Can be used multiple times.",
+    )
+    parser.add_argument(
+        "--item_type",
+        type=str,
+        default=None,
+        help="Use data_object or collection to show only items of that type. By default all items"
+        " are returned."
+    )
+
+    args = parser.parse_args()
+    with _cli_auth(ienv_path=_get_ienv_path()) as session:
+        ipath = _parse_remote(args.remote_path, session)
+        search_res = search_data(
+            session,
+            ipath,
+            path_pattern=args.path_pattern,
+            checksum=args.checksum,
+            metadata=args.metadata,
+            item_type=args.item_type,
+        )
+        for cur_path in search_res:
+            print(cur_path)
diff --git a/ibridges/icat_columns.py b/ibridges/icat_columns.py
index a11a3be3..586b211b 100644
--- a/ibridges/icat_columns.py
+++ b/ibridges/icat_columns.py
@@ -12,8 +12,10 @@
 DATA_CHECKSUM = imodels.DataObject.checksum
 META_COLL_ATTR_NAME = imodels.CollectionMeta.name
 META_COLL_ATTR_VALUE = imodels.CollectionMeta.value
+META_COLL_ATTR_UNITS = imodels.CollectionMeta.units
 META_DATA_ATTR_NAME = imodels.DataObjectMeta.name
 META_DATA_ATTR_VALUE = imodels.DataObjectMeta.value
+META_DATA_ATTR_UNITS = imodels.DataObjectMeta.units
 RESC_NAME = imodels.Resource.name
 RESC_PARENT = imodels.Resource.parent
 RESC_STATUS = imodels.Resource.status
diff --git a/ibridges/search.py b/ibridges/search.py
index 24fa6b21..25b4b934 100644
--- a/ibridges/search.py
+++ b/ibridges/search.py
@@ -2,19 +2,48 @@
 
 from __future__ import annotations
 
+from collections import namedtuple
 from typing import Optional, Union
 
 from ibridges import icat_columns as icat
 from ibridges.path import IrodsPath
 from ibridges.session import Session
 
+META_COLS = {
+    "collection": (icat.META_COLL_ATTR_NAME, icat.META_COLL_ATTR_VALUE, icat.META_COLL_ATTR_UNITS),
+    "data_object": (icat.META_DATA_ATTR_NAME, icat.META_DATA_ATTR_VALUE, icat.META_DATA_ATTR_UNITS)
+}
 
-def search_data(
+
+class MetaSearch(namedtuple("MetaSearch", ["key", "value", "units"], defaults=[..., ..., ...])):
+    """Named tuple to search for objects and collections.
+
+    The key, value and units default to the elipsis (...), which indicate that the search
+    accepts anything for this slot. This is principally the same as using the iRODS wildcard
+    '%' symbol except that during creation using elipses for key, value and units will raise
+    a ValueError. Note that the None value has a different meaning, where it will actually test
+    for the entry being None/empty.
+    """
+
+    def __new__(cls, key=..., value=..., units=...):
+        """Create a new MetaSearch object."""
+        if key is ... and value is ... and units is ...:
+            raise ValueError("Cannot create metasearch without specifying either key, value or "
+                             "units.")
+        key = "%" if key is ... else key
+        value = "%" if value is ... else value
+        units = "%" if units is ... else units
+        return super(MetaSearch, cls).__new__(cls, key, value, units)
+
+
+def search_data(  # pylint: disable=too-many-branches
     session: Session,
     path: Optional[Union[str, IrodsPath]] = None,
+    path_pattern: Optional[str] = None,
     checksum: Optional[str] = None,
-    key_vals: Optional[dict] = None,
-) -> list[dict]:
+    metadata: Union[None, MetaSearch, list[MetaSearch], list[tuple]] = None,
+    item_type: Optional[str] = None,
+) -> list[IrodsPath]:
     """Search for collections, data objects and metadata.
 
     By default all accessible collections and data objects are returned.
@@ -25,12 +54,24 @@ def search_data(
     ----------
     session:
         Session to search with.
-    path: str
-        (Partial) path or IrodsPath
-    checksum: str
-        (Partial) checksum
-    key_vals : dict
-        Attribute name mapping to values.
+    path:
+        IrodsPath to the collection to search into, collection itself will not be considered.
+        By default the home collection is searched.
+    path_pattern:
+        Search pattern in the path to look for. Allows for the '%' wildcard.
+        For example, use '%.txt' to look for all txt data objects.
+    checksum:
+        Checksum of the dataobject, wildcard '%' can be used. If left out, no checksum will be
+        matched.
+    metadata:
+        Metadata triples that constrain the key, value and units of the results.
+        For example, to get only items having a metadata entry with value "x",
+        use MetaSearch(value="x"). See :class:`MetaSearch` for more detail.
+        You can also provide a list of constraints. Then each of these constrains
+        will have to be satisfied for the data item to show up in the results.
+    item_type:
+        Type of the item to search for, by default None indicating both data objects and collections
+        are returned. Set to "data_object" for data objects and "collection" for collections.
 
     Raises
     ------
@@ -47,90 +88,145 @@ def search_data(
 
     Examples
     --------
-    >>> # Find all sub collections and data objects for a path
-    >>> search_data(session, path="/path/to/sub/col/%")
+    >>> # Find data objects and collections
+    >>> search_data(session, "/path/to/sub/col", path_pattern="somefile.txt")
+
+    >>> # Find data ending with .txt in your home and on a collection path with the substring "sub"
+    >>> search_data(session, path_pattern="%.txt")
+    >>> search_data(session, path_pattern="%sub/%.txt")
+
+    >>> # Find all data objects with a specific checksum in the home collection
+    >>> search_data(session, checksum="sha2:wW+wG+JxwHmE1uXEvRJQxA2nEpVJLRY2bu1KqW1mqEQ=")
+    [IrodsPath(/, somefile.txt), IrodsPath(/, someother.txt)]
 
-    >>> # Find all data objects with a specific checksum
-    >>> search_data(session, checksum="...")
+    >>> # Checksums can have wildcards as well, but beware of collisions:
+    >>> search_data(session, checksum="sha2:wW+wG%")
+    [IrodsPath(/, somefile.txt), IrodsPath(/, someother.txt)]
 
-    >>> # Find data objects and collections with some key/value pair.
-    >>> search_data(session, key_vals={"some_key": "some_value"})
+    >>> # Find data objects and collections with some metadata key
+    >>> search_data(session, metadata=MetaSearch(key="some_key"))
+
+    >>> # Search for data labeled with several metadata constraints
+    >>> search_data(session, metadata=[MetaSearch("some_key"), MetaSearch(value="other_value")]
+
+    >>> # Find data from metadata values using the wildcard
+    >>> # Will find all data and collections with e.g. "my_value" and "some_value"
+    >>> search_data(session, metadata=MetaSearch(value="%_value"))
+
+    >>> # Find data using metadata units
+    >>> search_data(session, metadata=MetaSearch(units="kg"))
+
+    >>> # Different conditions can be combined, only items for which all is True will be returned
+    >>> search_data(session, path_pattern="%.txt", metadata=MetaSearch(key="some_key", units="kg")
+
+    >>> # Find data without units
+    >>> search_data(session, metadata=MetaSearch(units=None)
+
+    >>> Find only data objects
+    >>> search_data(session, path_pattern="x%", item_type="data_object")
 
     """
-    if path is None and checksum is None and key_vals is None:
+    # Input validation
+    if path_pattern is None and checksum is None and metadata is None:
         raise ValueError(
             "QUERY: Error while searching in the metadata: No query criteria set."
-            + " Please supply either a path, checksum or key_vals."
+            + " Please supply either a path_pattern, checksum, key, value or units."
         )
+    if path is None:
+        path = session.home
+    path = IrodsPath(session, path)
+
+    if metadata is None:
+        metadata = []
+    if isinstance(metadata, MetaSearch):
+        metadata = [metadata]
 
-    # create the query for collections; we only want to return the collection name
-    coll_query = session.irods_session.query(icat.COLL_NAME)
-    # create the query for data objects; we need the collection name, the data name and its checksum
-    data_query = session.irods_session.query(icat.COLL_NAME, icat.DATA_NAME, icat.DATA_CHECKSUM)
-    data_name_query = session.irods_session.query(
-        icat.COLL_NAME, icat.DATA_NAME, icat.DATA_CHECKSUM
-    )
     # iRODS queries do not know the 'or' operator, so we need three searches
     # One for the collection, and two for the data
     # one data search in case path is a collection path and we want to retrieve all data there
     # one in case the path is or ends with a file name
-    path_params = _path_params(path)
-    if path_params:
-        path, name, parent = path_params
-        # all collections starting with path
-        coll_query = coll_query.filter(icat.LIKE(icat.COLL_NAME, path))
-
-        # all data objects in path
-        data_query = data_query.filter(icat.LIKE(icat.COLL_NAME, path))
-        # all data objects on path.parent with name
-        data_name_query = data_name_query.filter(icat.LIKE(icat.DATA_NAME, name)).filter(
-            icat.LIKE(icat.COLL_NAME, parent)
-        )
-    if key_vals:
-        for key in key_vals:
-            data_query.filter(icat.LIKE(icat.META_DATA_ATTR_NAME, key))
-            coll_query.filter(icat.LIKE(icat.META_COLL_ATTR_NAME, key))
-            data_name_query.filter(icat.LIKE(icat.META_DATA_ATTR_NAME, key))
-            if key_vals[key]:
-                data_query.filter(icat.LIKE(icat.META_DATA_ATTR_VALUE, key_vals[key]))
-                coll_query.filter(icat.LIKE(icat.META_COLL_ATTR_VALUE, key_vals[key]))
-                data_name_query = data_name_query.filter(
-                    icat.LIKE(icat.META_DATA_ATTR_VALUE, key_vals[key])
-                )
-
-    results = []
-    if checksum:
-        data_query = data_query.filter(icat.LIKE(icat.DATA_CHECKSUM, checksum))
-        data_name_query = data_name_query.filter(icat.LIKE(icat.DATA_CHECKSUM, checksum))
-    else:
-        # gather collection results
-        coll_res = list(coll_query.get_results())
-        if len(coll_res) > 0:
-            results.extend(coll_res)
+    queries = []
+    if item_type != "data_object" and checksum is None:
+        # create the query for collections; we only want to return the collection name
+        coll_query = session.irods_session.query(icat.COLL_NAME)
+        coll_query = coll_query.filter(icat.LIKE(icat.COLL_NAME, _postfix_wildcard(path)))
+        queries.append((coll_query, "collection"))
+    if item_type != "collection":
+        # create the query for data objects; we need the collection name, the data name and checksum
+        data_query = session.irods_session.query(icat.COLL_NAME, icat.DATA_NAME, icat.DATA_CHECKSUM)
+        data_query = data_query.filter(icat.LIKE(icat.COLL_NAME, _postfix_wildcard(path)))
+        queries.append((data_query, "data_object"))
+
+        data_name_query = session.irods_session.query(icat.COLL_NAME, icat.DATA_NAME,
+                                                    icat.DATA_CHECKSUM)
+        data_name_query.filter(icat.LIKE(icat.COLL_NAME, f"{path}"))
+        queries.append((data_name_query, "data_object"))
+
+    if path_pattern is not None:
+        _path_filter(path_pattern, queries)
+
+
+    for mf in metadata:
+        _meta_filter(mf, queries)
+
+    if checksum is not None:
+        _checksum_filter(checksum, queries)
+
+    query_results = []
+    for q in queries:
+        query_results.extend(list(q[0]))
 
     # gather results, data_query and data_name_query can contain the same results
-    results.extend([
+    results = [
         dict(s) for s in set(frozenset(d.items())
-                for d in list(data_query) + list(data_name_query))
-    ])
-
+                for d in query_results)
+    ]
     for item in results:
         if isinstance(item, dict):
             new_keys = [k.icat_key for k in item.keys()]
             for n_key, o_key in zip(new_keys, item.keys()):
                 item[n_key] = item.pop(o_key)
 
-    return results
+    # Convert the results to IrodsPath objects.
+    ipath_results = []
+    for res in results:
+        if "DATA_NAME" in res:
+            ipath_results.append(IrodsPath(session, res["COLL_NAME"], res["DATA_NAME"]))
+        else:
+            ipath_results.append(IrodsPath(session, res["COLL_NAME"]))
+    return ipath_results
 
-def _path_params(path_param):
-    """Parse the path parameter and return, path, name and parent."""
-    if path_param:
-        path = str(path_param)
-        if len(path.rsplit("/", maxsplit=1)) > 1:
-            parent = path.rsplit("/", maxsplit=1)[0]
-            name = path.rsplit("/", maxsplit=1)[1]
+
+def _prefix_wildcard(pattern):
+    if pattern.startswith("%"):
+        return pattern
+    return f"%/{pattern}"
+
+def _postfix_wildcard(path):
+    if str(path).endswith("/"):
+        return f"{path}%"
+    return f"{path}/%"
+
+def _path_filter(path_pattern, queries):
+    for q, q_type in queries:
+        if q_type == "collection":
+            q.filter(icat.LIKE(icat.COLL_NAME, _prefix_wildcard(path_pattern)))
         else:
-            name = path
-            parent = "%"
-        return path, name, parent
-    return None
+            split_pat = path_pattern.rsplit("/", maxsplit=1)
+            q.filter(icat.LIKE(icat.DATA_NAME, split_pat[-1]))
+
+            if len(split_pat) == 2:
+                q.filter(icat.LIKE(icat.COLL_NAME, _prefix_wildcard(split_pat[0])))
+
+
+def _meta_filter(metadata, queries):
+    if len(metadata) < 1 or len(metadata) > 3:
+        raise ValueError(f"Metadata contraints need 1-3 values, not {len(metadata)}")
+    for q, q_type in queries:
+        for i_elem, elem in enumerate(MetaSearch(*metadata)):
+            q.filter(icat.LIKE(META_COLS[q_type][i_elem], elem))
+
+def _checksum_filter(checksum, queries):
+    for q, q_type in queries:
+        if q_type == "data_object":
+            q.filter(icat.LIKE(icat.DATA_CHECKSUM, checksum))
diff --git a/tutorials/00-FirstSteps.ipynb b/tutorials/00-FirstSteps.ipynb
index 592408f0..95ed1ba3 100644
--- a/tutorials/00-FirstSteps.ipynb
+++ b/tutorials/00-FirstSteps.ipynb
@@ -144,7 +144,7 @@
    "source": [
     "# some stats\n",
     "print(home)\n",
-    "print(home.exists())\n",
+    "print(IrodsPath(session, home).exists())\n",
     "print(home.dataobject_exists())\n",
     "print(home.collection_exists())\n",
     "print(home.parent)\n",
@@ -224,7 +224,7 @@
    "outputs": [],
    "source": [
     "# create a local file\n",
-    "local_path = Path.expanduser(Path('~')).joinpath(\"demofile.txt\")\n",
+    "local_path = Path.home().joinpath(\"demofile.txt\")\n",
     "f = open(local_path, \"a\")\n",
     "f.write(\"My content! Super important\")\n",
     "f.close()"
@@ -268,7 +268,7 @@
     "# upload the file to our collection\n",
     "from pprint import pprint\n",
     "ops = upload(session, local_path, irods_path, dry_run=True, overwrite=True)\n",
-    "pprint(ops)"
+    "ops.print_summary()"
    ]
   },
   {
@@ -286,7 +286,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "upload(session, local_path, irods_path, overwrite=True)"
+    "changes = upload(session, local_path, irods_path, overwrite=True)\n",
+    "changes.print_summary()"
    ]
   },
   {
@@ -394,9 +395,10 @@
    "outputs": [],
    "source": [
     "from pathlib import Path\n",
-    "local_path = Path(\"~/Downloads\").expanduser()\n",
+    "local_path = Path.home().joinpath(\"Downloads\")\n",
     "print(\"Download folder exists\", local_path.is_dir())\n",
-    "download(session, ipath, local_path, overwrite=True)"
+    "local_changes = download(session, ipath, local_path, overwrite=True)\n",
+    "local_changes.print_summary()"
    ]
   },
   {
@@ -423,11 +425,12 @@
    "outputs": [],
    "source": [
     "# upload a file\n",
-    "local_path = Path(\"~\").expanduser().joinpath(\"demofile.txt\")\n",
+    "local_path = Path.home().joinpath(\"demofile.txt\")\n",
     "irods_path = IrodsPath(session, session.home, \"new_coll\")\n",
     "if not irods_path.collection_exists():\n",
     "    coll = IrodsPath.create_collection(session, irods_path)\n",
-    "upload(session, local_path, irods_path, overwrite = True)"
+    "changes = upload(session, local_path, irods_path, overwrite = True)\n",
+    "changes.print_summary()"
    ]
   },
   {
@@ -445,19 +448,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "obj = irods_path.joinpath(\"demofile.txt\").dataobject"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "2e8f3c90",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from ibridges.meta import MetaData\n",
-    "obj_meta = MetaData(obj)\n",
-    "print(obj_meta)"
+    "obj = irods_path.joinpath(\"demofile.txt\")\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -475,9 +467,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "obj_meta.add('NewKey', 'NewValue')\n",
-    "obj_meta.add('NewKey', 'AnotherValue')\n",
-    "print(obj_meta)"
+    "obj.meta.add('NewKey', 'NewValue')\n",
+    "obj.meta.add('NewKey', 'AnotherValue')\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -487,8 +479,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "obj_meta.delete('NewKey', 'NewValue')\n",
-    "print(obj_meta)"
+    "obj.meta.delete('NewKey', 'NewValue')\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -506,8 +498,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "obj_meta.set(\"NewKey\", \"YetAnotherValue\")\n",
-    "print(obj_meta)"
+    "obj.meta.set(\"NewKey\", \"YetAnotherValue\")\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -526,7 +518,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "for md in obj_meta:\n",
+    "for md in obj.meta:\n",
     "    print(md.name, \"has value\", md.value)"
    ]
   },
@@ -553,11 +545,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from ibridges.search import search_data\n",
+    "from ibridges.search import search_data, MetaSearch\n",
     "\n",
     "#all collections and objects with \n",
-    "key_vals = {\"NEWKEY\": \"YetAnotherValue\"}\n",
-    "result = search_data(session, key_vals=key_vals)\n",
+    "result = search_data(session, metadata=MetaSearch(key=\"NewKey\", value=\"YetAnotherValue\"))\n",
     "print(result)"
    ]
   },
@@ -566,9 +557,15 @@
    "id": "9aca8830",
    "metadata": {},
    "source": [
-    "The result gives us the collection, the object name, and its checksum.\n",
-    "\n",
-    "The wildcard in iRODS searches is the `%`."
+    "In the result we find the `IrodsPath` of the found collections and data objects."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "861e4b95",
+   "metadata": {},
+   "source": [
+    "If we are unsure about the exact metadata or collection or object name, we can use the wildcard `%`:"
    ]
   },
   {
@@ -578,7 +575,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "result = search_data(session, path=session.home+\"/new_coll/%\")\n",
+    "result = search_data(session, session.home, path_pattern=\"new_coll/%\")\n",
     "print(result)"
    ]
   },
@@ -766,7 +763,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.13"
+   "version": "3.9.18"
   }
  },
  "nbformat": 4,
diff --git a/tutorials/04-Metadata.ipynb b/tutorials/04-Metadata.ipynb
index 10005c2d..b8752841 100644
--- a/tutorials/04-Metadata.ipynb
+++ b/tutorials/04-Metadata.ipynb
@@ -49,6 +49,31 @@
     "session = interactive_auth()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "eb0524a2",
+   "metadata": {},
+   "source": [
+    "Make sure we have our *demo* collection and file available:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "19ad51aa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ibridges.path import IrodsPath\n",
+    "\n",
+    "irods_path = IrodsPath(session, '~')\n",
+    "print(\"Current working location:\", irods_path)\n",
+    "irods_coll_path = irods_path.joinpath('demo')\n",
+    "print(\"New collection name:\", irods_coll_path)\n",
+    "coll = IrodsPath.create_collection(session, irods_coll_path)\n",
+    "print(\"New collection is created:\", irods_coll_path.collection_exists())"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "81094d42",
@@ -65,13 +90,10 @@
    "outputs": [],
    "source": [
     "from ibridges.path import IrodsPath\n",
-    "from ibridges.meta import MetaData\n",
     "\n",
     "irods_coll_path = IrodsPath(session, '~').joinpath('demo')\n",
-    "obj = irods_coll_path.joinpath('demofile.txt').dataobject\n",
-    "\n",
-    "obj_metadata = MetaData(obj)\n",
-    "print(obj_metadata)"
+    "obj = irods_coll_path.joinpath('demofile.txt')\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -100,8 +122,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "obj_metadata.add('Key', 'Value', 'Units')\n",
-    "print(obj_metadata)"
+    "obj.meta.add('Key', 'Value', 'Units')\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -119,8 +141,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "obj_metadata.add('Author', 'Christine')\n",
-    "print(obj_metadata)"
+    "obj.meta.add('Author', 'Christine')\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -138,8 +160,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "obj_metadata.add('Author', 'Raoul')\n",
-    "print(obj_metadata)"
+    "obj.meta.add('Author', 'Raoul')\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -163,7 +185,7 @@
    "id": "cb5b7fef",
    "metadata": {},
    "source": [
-    "If you wish to *overwrite* a value, you can first add the new metadata key-value-units triple as above and subsequently remove the old one, you need to specify the whole triple if the metadata contains a units part. As you see the following command will fail:"
+    "If you wish to *overwrite* a value, you will first have to remove the old metadata and subsequently add a new metadata entry. **NOTE, that all entries with the key will be deleted.** If you want to be more specific you will need to give the value and the units."
    ]
   },
   {
@@ -173,25 +195,31 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "obj_metadata.delete('Key', 'Value')"
+    "obj.meta.delete('Author')\n",
+    "print(obj.meta)"
    ]
   },
   {
-   "cell_type": "markdown",
-   "id": "77993416",
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3348ce2d",
    "metadata": {},
+   "outputs": [],
    "source": [
-    "While this one will succeed:"
+    "obj.meta.add('Author', 'Raoul')\n",
+    "obj.meta.add('Author', 'Christine')\n",
+    "print(obj.meta)"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "ce98d47c",
+   "id": "6c0eaf42",
    "metadata": {},
    "outputs": [],
    "source": [
-    "obj_metadata.delete('Key', 'Value', 'Units')"
+    "obj.meta.delete('Author', 'Christine')\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -209,9 +237,50 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "print(obj_metadata)\n",
-    "obj_metadata.set('Author', 'Maarten')\n",
-    "print(obj_metadata)"
+    "obj.meta.set('Author', 'Maarten')\n",
+    "print(obj.meta)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1af4d99f",
+   "metadata": {},
+   "source": [
+    "iRODS metadata also has a an entry called `units`. The same principles that we showed above, i.e. having the same key-value pair with several units, deleting and setting values, apply to units."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6c8ee90c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "obj.meta.add('key', 'value', 'units1')\n",
+    "obj.meta.add('key', 'value', 'units2')\n",
+    "print(obj.meta)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d39bf4eb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "obj.meta.set('key', 'value', 'units3')\n",
+    "print(obj.meta)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "97b93549",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "obj.meta.delete('key', 'value')\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -237,9 +306,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "coll = irods_coll_path.collection\n",
-    "coll_metadata = MetaData(coll)\n",
-    "print(coll_metadata)"
+    "coll = irods_coll_path\n",
+    "print(coll.meta)"
    ]
   },
   {
@@ -249,8 +317,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "coll_metadata.add('TypeOfCollection', 'Results')\n",
-    "print(coll_metadata)"
+    "coll.meta.add('TypeOfCollection', 'Results')\n",
+    "print(coll.meta)"
    ]
   },
   {
@@ -279,11 +347,11 @@
    "outputs": [],
    "source": [
     "from datetime import datetime\n",
-    "coll_metadata.add('prov:wasGeneratedBy', 'Christine')\n",
-    "coll_metadata.add('CollectionType', 'testcollection')\n",
-    "obj_metadata.add('prov:SoftwareAgent', 'iRODS jupyter Tutorial')\n",
-    "obj_metadata.add('prov:wasGeneratedBy', 'Maarten')\n",
-    "obj_metadata.add('DataType', 'testdata')"
+    "coll.meta.add('prov:wasGeneratedBy', 'Christine')\n",
+    "coll.meta.add('CollectionType', 'testcollection')\n",
+    "obj.meta.add('prov:SoftwareAgent', 'iRODS jupyter Tutorial')\n",
+    "obj.meta.add('prov:wasGeneratedBy', 'Maarten')\n",
+    "obj.meta.add('DataType', 'testdata')"
    ]
   },
   {
@@ -301,8 +369,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "print(coll_metadata)\n",
-    "print(obj_metadata)"
+    "print(coll.meta)\n",
+    "print()\n",
+    "print(obj.meta)"
    ]
   },
   {
@@ -318,9 +387,9 @@
    "id": "59518813",
    "metadata": {},
    "source": [
-    "Metadata does not only help you to. keep an overview over your data, but can also be used to select and retrieve data. In iBridges you can use the user-defined metadata and some system metadata fields to search for data.\n",
+    "Metadata does not only help you to keep an overview over your data, but can also be used to select and retrieve data. In iBridges you can use the user-defined metadata and some system metadata fields to search for data.\n",
     "\n",
-    "In our first example we list all collections and data objects in our iRODS home."
+    "In our first example, we are looking for objects and collections called *demo* in our `home`:"
    ]
   },
   {
@@ -330,27 +399,18 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from ibridges.search import search_data\n",
-    "result = search_data(session, path=session.home)\n",
+    "from ibridges.search import search_data, MetaSearch\n",
+    "result = search_data(session, path=session.home, path_pattern=\"demo\")\n",
     "print(result)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "31c88425",
+   "id": "ea7129f8",
    "metadata": {},
    "source": [
-    "The output is a list of Python dictionaries, where each dictionary contains\n",
-    "\n",
-    "1) Collections: `'COLL_NAME': '/<ZONE>/home/<YOUR PATH>'}`\n",
-    "2) Data Objects: \n",
-    "    ```\n",
-    "    {'COLL_NAME': '/<ZONE>/home/<YOUR PATH>', \n",
-    "     'DATA_NAME': '<OBJECT NAME>', \n",
-    "     'D_DATA_CHECKSUM': '<CHECKSUM>'}\n",
-    "    ```\n",
-    "\n",
-    "Now let's try to find data by its metadata. We will have to create a Python dictionary with the metadata keys and the metadata values as search criterion:"
+    "The output is a list of IrodsPath's indicating the locations of the data objects and collections.\n",
+    "If no `path` is provided, *ibridges* will automatically fall back on your `home`."
    ]
   },
   {
@@ -360,8 +420,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "key_vals = {'prov:wasGeneratedBy': 'Christine'}\n",
-    "result = search_data(session, key_vals=key_vals)\n",
+    "result = search_data(session, metadata=MetaSearch(key='prov:wasGeneratedBy', value='Christine'))\n",
     "print(result)"
    ]
   },
@@ -370,7 +429,7 @@
    "id": "81d73352",
    "metadata": {},
    "source": [
-    "If we do not want to specify the particular value of the key, we can use a *wildcard*. In iRODS the wildcard is `%`."
+    "If we do not want to specify the particular value for this metadata entry, we can leave it out."
    ]
   },
   {
@@ -380,8 +439,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "key_vals = {'prov:wasGeneratedBy': '%'}\n",
-    "result = search_data(session, key_vals=key_vals)\n",
+    "result = search_data(session, metadata=MetaSearch(key='prov:wasGeneratedBy'))\n",
     "print(result)"
    ]
   },
@@ -402,11 +460,200 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "key_vals = {'prov:wasGeneratedBy': '%'}\n",
-    "result = search_data(session, path='/nluu12p/home/research-test-christine/%', key_vals=key_vals)\n",
+    "result = search_data(session, path=IrodsPath(session, session.home, 'demo'),\n",
+    "                     metadata=MetaSearch(key='prov:wasGeneratedBy'))\n",
+    "print(result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8750ba85",
+   "metadata": {},
+   "source": [
+    "## Searches using wildcards"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5ec06eaa",
+   "metadata": {},
+   "source": [
+    "Sometimes we are not sure about the exact pattern that we search for, be it metadata keys, values and units or path patterns. iRODS knows the `%` sign as a wild card."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2d26000f",
+   "metadata": {},
+   "source": [
+    "### Wildcards in metadata"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4a7b1fe9",
+   "metadata": {},
+   "source": [
+    "Assume we know that some data was annotated according to the Prov-O Ontology and its abbrviation is `prov`, but we do not know which terms of that Ontology was used. In such a case we can find all metadata annotated with a key with the prefix `prov:` like this:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e19e205c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = search_data(session, path=IrodsPath(session, session.home),\n",
+    "                     metadata=MetaSearch(key='prov:%'))\n",
+    "print(result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ea29f04f",
+   "metadata": {},
+   "source": [
+    "### Wildcards in path patterns"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "efc56727",
+   "metadata": {},
+   "source": [
+    "Let us go back to the very first example of this section, we are loking for collections and data objects called `demo` and they need to lie directly in our `home`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e9c5636d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = search_data(session, path=session.home, path_pattern=\"demo\")\n",
     "print(result)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "edef8def",
+   "metadata": {},
+   "source": [
+    "How can we retrieve all `demo` collections and objects even if they are or lie in subcollections? Lets first create subcollections in `demo` called `demo` and `demo1`. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d1436239",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "irods_path = IrodsPath(session, \"demo\", \"demo\")\n",
+    "print(irods_path)\n",
+    "IrodsPath.create_collection(session, irods_path)\n",
+    "irods_path = IrodsPath(session, \"demo\", \"demo1\")\n",
+    "print(irods_path)\n",
+    "IrodsPath.create_collection(session, irods_path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dcd5849c",
+   "metadata": {},
+   "source": [
+    "Now let's see how to use the wildcard to find those two collections."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "776e36f0",
+   "metadata": {},
+   "source": [
+    "#### 1. Find all data and collections ending with `demo`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2354ae2d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = search_data(session, path=session.home, path_pattern=\"%demo\")\n",
+    "print('\\n'.join([str(p) for p in result]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3c710e35",
+   "metadata": {},
+   "source": [
+    "#### 2. Find all data and collections starting with `demo`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e062cbf1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = search_data(session, path=session.home, path_pattern=\"demo%\")\n",
+    "print('\\n'.join([str(p) for p in result]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "05958fd7",
+   "metadata": {},
+   "source": [
+    "#### 3. Find all collections and data called `demo` that on the 5th layer of the collection tree"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "26cbab9d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = search_data(session, path=session.home, path_pattern=\"%/%/%/%/%/demo\")\n",
+    "print('\\n'.join([str(p) for p in result]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4972990d",
+   "metadata": {},
+   "source": [
+    "#### 4. Find all `txt` files that lie on a collection path that contains `demo`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "70d1d9ff",
+   "metadata": {},
+   "source": [
+    "For this case we have to think of a pattern for the collection path and the object name and separate both with `/`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e6781e26",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "coll_pattern = \"%demo%\"\n",
+    "obj_pattern = \"%.txt\"\n",
+    "print(f\"Search pattern: {coll_pattern+'/'+obj_pattern}\")\n",
+    "result = search_data(session, path=session.home, path_pattern=coll_pattern+\"/\"+obj_pattern)\n",
+    "print('\\n'.join([str(p) for p in result]))"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "14bc70b0",
@@ -420,7 +667,7 @@
    "id": "946e355c",
    "metadata": {},
    "source": [
-    "Now that we have the search results in a list of Python dictionaries, we can use the information to create the full iRODS paths and continue working with them e.g. download them."
+    "Now that we have the search results we can use the `IrodsPath` to download them or to fetch more information:"
    ]
   },
   {
@@ -430,10 +677,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from ibridges.path import IrodsPath\n",
-    "\n",
-    "paths = [IrodsPath(session, r.get('COLL_NAME', '')).joinpath(r.get('DATA_NAME', '')) for r in result]\n",
-    "print(paths)"
+    "print(result[0].size)\n",
+    "print(result[0].collection_exists())\n",
+    "print(result[0].dataobject_exists())"
    ]
   },
   {
@@ -456,17 +702,35 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "id": "6a92e519",
    "metadata": {},
    "outputs": [],
    "source": [
     "from ibridges.data_operations import create_meta_archive\n",
     "\n",
-    "collection_path = IrodsPath(session, \"~\", \"Demo\")\n",
+    "collection_path = IrodsPath(session, \"demo\")\n",
     "create_meta_archive(session, collection_path, \"meta_archive.json\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "8c20eaee",
+   "metadata": {},
+   "source": [
+    "This creates a file \"meta_archive.json\" in your current local directory of this jupyter notebook which contains all metadata of all subcollections and data objects in this collection \"demo\"."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3c378d37",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!cat meta_archive.json"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "00d0e863",
@@ -506,7 +770,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.6"
+   "version": "3.9.18"
   }
  },
  "nbformat": 4,
diff --git a/tutorials/07-Streaming-iRODS-data.ipynb b/tutorials/07-Streaming-iRODS-data.ipynb
index f7dc32e8..32bd7408 100644
--- a/tutorials/07-Streaming-iRODS-data.ipynb
+++ b/tutorials/07-Streaming-iRODS-data.ipynb
@@ -37,30 +37,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "id": "9f19a37e",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "/nluu12p/home/research-test-christine/my_books:\n",
-      "  /nluu12p/home/research-test-christine/my_books/AdventuresSherlockHolmes.txt\n",
-      "  /nluu12p/home/research-test-christine/my_books/AliceAdventuresInWonderLand.txt\n",
-      "  /nluu12p/home/research-test-christine/my_books/DonQuixote.txt\n",
-      "  /nluu12p/home/research-test-christine/my_books/Dracula.txt\n",
-      "  /nluu12p/home/research-test-christine/my_books/Frankenstein.txt\n",
-      "  /nluu12p/home/research-test-christine/my_books/Phantasmagoria.txt\n",
-      "  /nluu12p/home/research-test-christine/my_books/RobinsonCrusoe.txt\n",
-      "  /nluu12p/home/research-test-christine/my_books/TheHuntingOfTheSnark.txt\n",
-      "  /nluu12p/home/research-test-christine/my_books/ThroughTheLookingGlass.txt\n",
-      "  /nluu12p/home/research-test-christine/my_books/TravelsIntoSeveralRemoteNationsOfTheWorld.txt\n",
-      "  /nluu12p/home/research-test-christine/my_books/TwentyThousandLeaguesUnderTheSea.txt\n",
-      "\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "!ibridges list irods:my_books"
    ]
@@ -82,7 +62,7 @@
    "source": [
     "from pprint import pprint\n",
     "from ibridges.interactive import interactive_auth\n",
-    "from ibridges import search_data"
+    "from ibridges.search import search_data, MetaSearch"
    ]
   },
   {
@@ -113,7 +93,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "data = search_data(session, key_vals={KEY: VALUE})\n",
+    "data = search_data(session, metadata=MetaSearch(key=KEY, value=VALUE))\n",
     "pprint(data)"
    ]
   },
@@ -153,8 +133,7 @@
    "outputs": [],
    "source": [
     "for item in data:\n",
-    "    irods_path = IrodsPath(session, item['COLL_NAME'], item['DATA_NAME'])\n",
-    "    with irods_path.dataobject.open('r') as objRead:\n",
+    "    with item.dataobject.open('r') as objRead:\n",
     "        text = text + objRead.read().decode()"
    ]
   },
@@ -345,7 +324,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.13"
+   "version": "3.9.18"
   }
  },
  "nbformat": 4,


Test patch:
diff --git a/.github/workflows/integration-tests-yoda.yml b/.github/workflows/integration-tests-yoda.yml
index 012412c0..5005f076 100644
--- a/.github/workflows/integration-tests-yoda.yml
+++ b/.github/workflows/integration-tests-yoda.yml
@@ -32,7 +32,7 @@ jobs:
 
     - name: Clone Yoda repo for Docker Setup
       run: |
-        git clone -b release-1.9 --single-branch https://github.com/UtrechtUniversity/yoda.git
+        git clone -b v1.9.4 --single-branch https://github.com/UtrechtUniversity/yoda.git
 
     - name: Prepare hosts file for integration tests
       run: |
diff --git a/docker/irods_client/tests/conftest.py b/docker/irods_client/tests/conftest.py
index 383e8414..30007937 100644
--- a/docker/irods_client/tests/conftest.py
+++ b/docker/irods_client/tests/conftest.py
@@ -51,7 +51,9 @@ def testdata():
 
 @pytest.fixture(scope="session")
 def collection(session):
-    return create_collection(session, IrodsPath(session, "~", "test_collection"))
+    coll = create_collection(session, IrodsPath(session, "~", "test_collection"))
+    yield coll
+    IrodsPath(session, coll.path).remove()
 
 
 @pytest.fixture(scope="session")
diff --git a/docker/irods_client/tests/test_cli.py b/docker/irods_client/tests/test_cli.py
index 7a9ef4a1..1a08f3fd 100644
--- a/docker/irods_client/tests/test_cli.py
+++ b/docker/irods_client/tests/test_cli.py
@@ -4,6 +4,7 @@
 from pathlib import Path
 
 import pytest
+from pytest import mark
 
 from ibridges import IrodsPath
 from ibridges.data_operations import create_collection
@@ -20,7 +21,8 @@ def pass_opts(config):
             or config.get("has_cached_pw", False)):
         pass_opts = {}
     else:
-        pass_opts = {"input": config["password"].encode()}
+        pass_opts = {"input": config["password"]}
+    pass_opts["text"] = True
     pass_opts["check"] = True
     return pass_opts
 
@@ -71,6 +73,7 @@ def test_upload_download_cli(session, config, testdata, tmpdir, irods_env_file,
 
     _check_files_equal(testdata/"plant2.rtf", testdata/"plant.rtf")
     (testdata/"plant2.rtf").unlink()
+    ipath.remove()
 
     # Upload a directory and check if the dataobjects and collections are created.
     ipath = IrodsPath(session, "~", "test")
@@ -91,6 +94,8 @@ def test_upload_download_cli(session, config, testdata, tmpdir, irods_env_file,
     subprocess.run(["ibridges", "sync", "irods:~/test/testdata", tmpdir], **pass_opts)
     for fname in testdata.glob("*"):
         assert _check_files_equal(testdata/fname.name, tmpdir/fname.name)
+    ipath.remove()
+
 
 def test_upload_download_metadata(session, config, testdata, tmpdir, irods_env_file, pass_opts):
     ipath_collection = IrodsPath(session, "meta_test")
@@ -125,8 +130,47 @@ def test_upload_download_metadata(session, config, testdata, tmpdir, irods_env_f
     ipath_collection.remove()
 
 
-def test_list_cli(config, pass_opts, irods_env_file):
+def test_list_cli(config, pass_opts, irods_env_file, collection):
     # Check if the listing works without any errors.
     subprocess.run(["ibridges", "init", irods_env_file], **pass_opts)
     subprocess.run(["ibridges", "list"], **pass_opts)
-    subprocess.run(["ibridges", "list", "irods:test"], **pass_opts)
+    subprocess.run(["ibridges", "list", f"irods:{collection.path}"], **pass_opts)
+
+
+@mark.parametrize(
+    "search,nlines",
+    [
+        (["--path-pattern", "test_search"], 1),
+        (["--path-pattern", "test_search2"], 0),
+        (["--path-pattern", "test_search/%"], 6),
+        (["--path-pattern", "test_search/%/%"], 1),
+        (["--path-pattern", "%.rtf"], 4),
+        (["--checksum", "sha2:uJzC+gqi59rVJu8PoBAaTstNUUnFMxW9HsJzsJUb1ao="], 1),
+        (["--checksum", "sha2:uJzC+gqi5%"], 1),
+        (["--path-pattern", "%", "--item_type", "data_object"], 6),
+        (["--path-pattern", "%", "--item_type", "collection"], 2),
+        (["--path-pattern", "%/%", "--item_type", "collection"], 2),
+        (["--metadata", "search"], 1),
+        (["--metadata", "search", "sval", "kg"], 1),
+        (["--metadata", "search", "--metadata", "search2"], 1),
+        (["--metadata", "search", "--metadata", "does not exist"], 0),
+    ]
+)
+def test_search_cli(session, config, pass_opts, irods_env_file, testdata, search, nlines):
+    subprocess.run(["ibridges", "init", irods_env_file], **pass_opts)
+    ipath_coll = IrodsPath(session, "test_search_x", "test_search")
+    IrodsPath.create_collection(session, ipath_coll)
+    subprocess.run(["ibridges", "sync", testdata, "irods:test_search_x/test_search"], **pass_opts)
+    assert ipath_coll.collection_exists()
+    ipath_coll.meta.clear()
+    ipath_coll.meta.add("search", "sval", "kg")
+    ipath_coll.meta.add("search2", "small")
+
+    ret = subprocess.run(["ibridges", "search", "irods:test_search_x", *search], capture_output=True,
+                         **pass_opts)
+    stripped_str = ret.stdout.strip("\n")
+    if stripped_str == "":
+        assert nlines == 0
+    else:
+        assert len([x for x in stripped_str.split("\n") if not x.startswith("Your iRODS")]) == nlines
+    ipath_coll.remove()
diff --git a/docker/irods_client/tests/test_search.py b/docker/irods_client/tests/test_search.py
new file mode 100644
index 00000000..8005fa58
--- /dev/null
+++ b/docker/irods_client/tests/test_search.py
@@ -0,0 +1,60 @@
+from pytest import mark
+
+from ibridges import IrodsPath, search_data, upload
+from ibridges.search import MetaSearch
+
+
+def _found(search_res, path):
+    for res in search_res:
+        assert isinstance(res, IrodsPath)
+        if str(res) == str(path):
+            return True
+    return False
+
+@mark.parametrize("item_name", ["collection", "dataobject"])
+def test_find_path(session, item_name, request):
+    item = request.getfixturevalue(item_name)
+    ipath = IrodsPath(session, item.path)
+    assert _found(search_data(session, path_pattern=item.name), item.path)
+    assert _found(search_data(session, path_pattern=item.name[:4]+"%"), item.path)
+    assert _found(search_data(session, ipath.parent, item.name), item.path)
+    assert _found(search_data(session, ipath.parent.parent, item.name), item.path)
+    assert _found(search_data(session, IrodsPath(session, "/"), item.name), item.path)
+    pat = item.name[:4] + "%"
+    assert _found(search_data(session, path_pattern=pat), item.path)
+    pat = "%" + item.name[-3:]
+    assert _found(search_data(session, path_pattern=pat), item.path)
+    pat = f"{ipath.parent.name}/%{ipath.name[2:]}"
+    assert _found(search_data(session, IrodsPath(session, "/"), path_pattern=pat), item.path)
+    assert not _found(search_data(session, path_pattern="random_file"), item.path)
+
+
+def test_find_checksum(session, dataobject):
+    ipath = IrodsPath(session, dataobject.path)
+    checksum = ipath.checksum
+    assert _found(search_data(session, IrodsPath(session, "/"), checksum=checksum), ipath)
+    assert not _found(search_data(session, IrodsPath(session, "/"), checksum="sha2:a9s8d7hjas"), ipath)
+    assert _found(search_data(session, IrodsPath(session, "/"), checksum=checksum[:15] + "%"), ipath)
+
+
+@mark.parametrize("metadata,is_found", [
+    (MetaSearch("Author", "Ben"), True),
+    (MetaSearch(units="kg"), True),
+    (MetaSearch(value="10"), True),
+    (MetaSearch(key="Random"), False),
+    (MetaSearch(key="Author", value="10"), False),
+    (MetaSearch(key="Author", units="kg"), False),
+    (MetaSearch(key="Mass", units=None), False),
+    ([MetaSearch(key="Author"), MetaSearch(value="10")], True),
+    ([MetaSearch("Author"), MetaSearch("Random")], False),
+])
+@mark.parametrize("item_name", ["collection", "dataobject"])
+def test_find_meta(session, item_name, request, metadata, is_found):
+    item = request.getfixturevalue(item_name)
+    ipath = IrodsPath(session, item.path)
+    ipath.meta.clear()
+    ipath.meta.add("Author", "Ben")
+    ipath.meta.add("Mass", "10", "kg")
+
+    res = _found(search_data(session, metadata=metadata), ipath)
+    assert res == is_found