Repository: bentoml/BentoML
Pull request: [Enhancement/CLI]Add "all" option for bento service delete
<!--- Thanks for sending a pull request! Please make sure to read the contribution guidelines, then fill out the blanks below before requesting a code review. -->

## Description
<!--- Describe your changes in detail -->
<!--- Attach screenshots here if appropriate. -->
Add `all` option for bento service delete.

```python
yc.repository.delete(all, labels, bento_tag, bento_name, bento_version)

"""
Params:
all: boolean, Set true to delete all bento services
bento_tag: Bento tag
labels: string, label selector to filter bento services to delete
bento_name: string 
bento_version: string, 
"""
```

Update `bentoml delete` CLI

```bash
$ bentoml delete --all --tag bento_tag,bento_tag --labels LABEL_SELECTOR --bento-name NAME --bento-version VERSION --yes
```

**Breaking change**

BentoML CLI no longer supports tags as arguments.  Tags are now an option for the `bentoml delete` command.

## Motivation and Context
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->
<!--- If it is based on a conversation in slack channel, pls quote related messages here -->

## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->

## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [x] New feature and improvements (non-breaking change which adds/improves functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Code Refactoring (internal change which is not user facing)
- [ ] Documentation
- [ ] Test, CI, or build
<!--- [ ] Others: describe the type of change if it does not fit to categories listed -->

## Component(s) if applicable
- [ ] BentoService (service definition, dependency management, API input/output adapters)
- [ ] Model Artifact (model serialization, multi-framework support)
- [ ] Model Server (mico-batching, dockerisation, logging, OpenAPI, instruments)
- [x] YataiService gRPC server (model registry, cloud deployment automation)
- [ ] YataiService web server (nodejs HTTP server and web UI)
- [ ] Internal (BentoML's own configuration, logging, utility, exception handling)
- [x] BentoML CLI

## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
<!--- If you plan to update documentation or tests in follow-up, please note -->
- [ ] My code follows the bentoml code style, both `./dev/format.sh` and
  `./dev/lint.sh` script have passed
  ([instructions](https://github.com/bentoml/BentoML/blob/master/DEVELOPMENT.md#style-check-and-auto-formatting-your-code)).
- [ ] My change reduces project test coverage and requires unit tests to be added
- [ ] I have added unit tests covering my code change
- [ ] My change requires a change to the documentation
- [ ] I have updated the documentation accordingly


Problem statement:
Add `bentoml prune` command for deleting all saved bundles
**Is your feature request related to a problem? Please describe.**
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->


**Describe the solution you'd like**
<!-- A clear and concise description of what you want to happen. -->
```
# remove all local saved bundle
bentoml prune --local

# remove in remote yatai service
bentoml prune --yatai-url=....

# confirm deleting interactively
bentoml prune --interactive
> Remove service IrisClassifer:V1231231? enter y(yes) or n(no):
> yes
> ..

```

**Describe alternatives you've considered**
<!-- A clear and concise description of any alternative solutions or features you've considered. -->

**Additional context**
<!-- Add any other context or screenshots about the feature request here. -->

Hints from issue discussion:
@parano What is the intended distinction between prune and `bento delete`? 
@tahentx `bentoml delete` currently can only delete one saved bundle at a time, this issue was meant for adding support for deleting in batches by labels, service name or DateTime range, and there's a python API for it right now called `prune`: https://docs.bentoml.org/en/latest/api/yatai_client.html#bentoml.yatai.client.bento_repository_api.BentoRepositoryAPIClient.prune

Great question, I actually think merging `delete` and `prune` into one API and one command make sense, e.g.:

```
yatai_client = get_yatai_client()

# delete single item
yatai_client.repository.delete('my_service:version')

# batch delete
yatai_client.repository.delete(labels='ci=failed')
yatai_client.repository.delete(name='my_service')
yatai_client.repository.delete(prune=True) # delete all

```

cc @yubozhao 
I think this is a great suggestion. We should combine the APIs together and make it easier to understand.

Commit message: formatting

Patch to review:
diff --git a/bentoml/cli/bento_management.py b/bentoml/cli/bento_management.py
index 2eb888340f5..d86b525168f 100644
--- a/bentoml/cli/bento_management.py
+++ b/bentoml/cli/bento_management.py
@@ -16,10 +16,7 @@
 from tabulate import tabulate
 
 from bentoml.utils.lazy_loader import LazyLoader
-from bentoml.cli.click_utils import (
-    _echo,
-    parse_bento_tag_list_callback,
-)
+from bentoml.cli.click_utils import _echo, parse_bento_tag_list_callback
 from bentoml.cli.utils import (
     human_friendly_age_from_datetime,
     _format_labels_for_print,
@@ -175,10 +172,30 @@ def list_bentos(
         _print_bentos_info(result, output)
 
     @cli.command(
-        help='Delete bento. To delete multiple bentos provide the name '
-        'version tag separated by "," for example "bentoml delete name:v1,name:v2',
+        help='Delete bentos. To delete a bento service use "--bento-name" and '
+        '"--bento-version" option: "bentoml delete --bento-name IrisClassifier '
+        '--bento-version 0.1.0". To delete multiple bentos, use "--bento-name" and/or '
+        '"--labels" to filter. To delete all bento services use "--all" option'
+    )
+    @click.option(
+        '--all', is_flag=True, help='Use this flag to remove all BentoServices'
+    )
+    @click.option(
+        '--tag',
+        type=click.STRING,
+        help='Bento tags. To delete multiple bentos provide the name version tag '
+        'separated by "," for example "bentoml delete --tag name:v1,name:v2',
+        callback=parse_bento_tag_list_callback,
+    )
+    @click.option(
+        '--labels',
+        type=click.STRING,
+        help="Label query to filter BentoServices, supports '=', '!=', 'IN', 'NotIn', "
+        "'Exists', and 'DoesNotExist'. (e.g. key1=value1, key2!=value2, key3 "
+        "In (value3, value3a), key4 DoesNotExist)",
     )
-    @click.argument("bentos", type=click.STRING, callback=parse_bento_tag_list_callback)
+    @click.option("--name", type=click.STRING, help='BentoService name')
+    @click.option("--version", type=click.STRING, help='BentoService version')
     @click.option(
         '--yatai-url',
         type=click.STRING,
@@ -186,28 +203,49 @@ def list_bentos(
         '"--yatai-url http://localhost:50050"',
     )
     @click.option(
-        '-y', '--yes', '--assume-yes', is_flag=True, help='Automatic yes to prompts'
+        '-y',
+        '--yes',
+        '--assume-yes',
+        is_flag=True,
+        help='Skip confirmation when deleting specific BentoService bundle',
     )
-    def delete(bentos, yatai_url, yes):
-        """Delete saved BentoService.
+    def delete(
+        all,  # pylint: disable=redefined-builtin
+        tag,
+        labels,
+        name,
+        version,
+        yatai_url,
+        yes,
+    ):
+        """Delete saved BentoServices.
 
         BENTO is the target BentoService to be deleted, referenced by its name and
         version in format of name:version. For example: "iris_classifier:v1.2.0"
 
         `bentoml delete` command also supports deleting multiple saved BentoService at
-        once, by providing name version tag separated by ",", for example:
-
-        `bentoml delete iris_classifier:v1.2.0,my_svc:v1,my_svc2:v3`
+        once, by providing either the BentoService name and/or labels
         """
-        for bento in bentos:
-            if not yes and not click.confirm(
-                f'Are you sure about delete {bento}? This will delete the BentoService '
-                f'saved bundle files permanently'
-            ):
-                return
-            yc = get_yatai_client(yatai_url)
-            yc.repository.delete(bento)
-            _echo(f'BentoService {bento} deleted')
+        yc = get_yatai_client(yatai_url)
+        # Backward compatible with the previous CLI, allows deletion with tag/s
+        if len(tag) > 0:
+            for item in tag:
+                yc.repository.delete(
+                    prune=all,
+                    labels=labels,
+                    bento_tag=item,
+                    bento_name=name,
+                    bento_version=version,
+                    require_confirm=False if yes else True,
+                )
+        else:
+            yc.repository.delete(
+                prune=all,
+                labels=labels,
+                bento_name=name,
+                bento_version=version,
+                require_confirm=False if yes else True,
+            )
 
     @cli.command(help='Pull BentoService from remote yatai server',)
     @click.argument("bento", type=click.STRING)
diff --git a/bentoml/yatai/client/bento_repository_api.py b/bentoml/yatai/client/bento_repository_api.py
index 91cb6766112..492b2377c98 100644
--- a/bentoml/yatai/client/bento_repository_api.py
+++ b/bentoml/yatai/client/bento_repository_api.py
@@ -18,6 +18,8 @@
 import os
 import logging
 import tarfile
+
+import click
 import requests
 import shutil
 
@@ -342,65 +344,113 @@ def list(
             raise BentoMLException(f'{error_code}:{error_message}')
         return result.bentos
 
-    def delete(self, bento):
-        """
-        Delete bento
-
-        Args:
-            bento: a BentoService identifier in the format of NAME:VERSION
-
-        Example:
-        >>>
-        >>> yatai_client = get_yatai_client()
-        >>> yatai_client.repository.delete('my_service:version')
-        """
-        track('py-api-delete')
-        if ':' not in bento:
-            raise BentoMLException(
-                'BentoService name or version is missing. Please provide in the '
-                'format of name:version'
-            )
-        name, version = bento.split(':')
+    def _delete_bento_bundle(self, bento_tag, require_confirm):
+        bento_pb = self.get(bento_tag)
+        if require_confirm and not click.confirm(f'Permanently delete {bento_tag}?'):
+            return
         result = self.yatai_service.DangerouslyDeleteBento(
-            DangerouslyDeleteBentoRequest(bento_name=name, bento_version=version)
+            DangerouslyDeleteBentoRequest(
+                bento_name=bento_pb.name, bento_version=bento_pb.version
+            )
         )
+
         if result.status.status_code != yatai_proto.status_pb2.Status.OK:
             error_code, error_message = status_pb_to_error_code_and_message(
                 result.status
             )
-            raise BentoMLException(
-                f'Failed to delete Bento {bento} {error_code}:{error_message}'
+            # Rather than raise Exception, continue to delete the next bentos
+            logger.error(
+                f'Failed to delete {bento_pb.name}:{bento_pb.version} - '
+                f'{error_code}:{error_message}'
             )
+        else:
+            logger.info(f'Deleted {bento_pb.name}:{bento_pb.version}')
 
-    def prune(self, bento_name=None, labels=None):
+    def delete(
+        self,
+        bento_tag=None,
+        labels=None,
+        bento_name=None,
+        bento_version=None,
+        prune=False,  # pylint: disable=redefined-builtin
+        require_confirm=False,
+    ):
         """
-        Delete all BentoServices that matches the specified criteria
+        Delete bentos that matches the specified criteria
 
         Args:
-            bento_name: optional
-            labels: optional
-
+            bento_tag: string
+            labels: string
+            bento_name: string
+            bento_version: string
+            prune: boolean, Set True to delete all BentoService
+            require_confirm: boolean
         Example:
-
+        >>>
         >>> yatai_client = get_yatai_client()
         >>> # Delete all bento services
-        >>> yatai_client.repository.prune()
-        >>> # Delete bento services that matches with the label `ci=failed`
-        >>> yatai_client.repository.prune(labels='ci=failed')
+        >>> yatai_client.repository.delete(prune=True)
+        >>> # Delete bento service with name is `IrisClassifier` and version `0.1.0`
+        >>> yatai_client.repository.delete(
+        >>>     bento_name='IrisClassifier', bento_version='0.1.0'
+        >>> )
+        >>> # or use bento tag
+        >>> yatai_client.repository.delete('IrisClassifier:v0.1.0')
+        >>> # Delete all bento services with name 'MyService`
+        >>> yatai_client.repository.delete(bento_name='MyService')
+        >>> # Delete all bento services with labels match `ci=failed` and `cohort=20`
+        >>> yatai_client.repository.delete(labels='ci=failed, cohort=20')
         """
-        track('py-api-prune')
-        list_bentos_result = self.list(bento_name=bento_name, labels=labels,)
-        if list_bentos_result.status.status_code != yatai_proto.status_pb2.Status.OK:
-            error_code, error_message = status_pb_to_error_code_and_message(
-                list_bentos_result.status
+        track('py-api-delete')
+
+        delete_list_limit = 50
+
+        if (
+            bento_tag is not None
+            and bento_name is not None
+            and bento_version is not None
+        ):
+            raise BentoMLException('Too much arguments')
+
+        if bento_tag is not None:
+            logger.info(f'Deleting saved Bento bundle {bento_tag}')
+            return self._delete_bento_bundle(bento_tag, require_confirm)
+        elif bento_name is not None and bento_tag is not None:
+            logger.info(f'Deleting saved Bento bundle {bento_name}:{bento_version}')
+            return self._delete_bento_bundle(
+                f'{bento_name}:{bento_version}', require_confirm
             )
-            raise BentoMLException(f'{error_code}:{error_message}')
-        for bento in list_bentos_result.bentos:
-            bento_tag = f'{bento.name}:{bento.version}'
-            try:
-                self.delete(bento_tag)
-            except BentoMLException as e:
-                logger.error(f'Failed to delete Bento {bento_tag}: {e}')
+        else:
+            # list of bentos
+            if prune is True:
+                logger.info('Deleting all BentoML saved bundles.')
+                # ignore other fields
+                bento_name = None
+                labels = None
+            else:
+                log_message = 'Deleting saved Bento bundles'
+                if bento_name is not None:
+                    log_message += f' with name: {bento_name},'
+                if labels is not None:
+                    log_message += f' with labels match to {labels}'
+                logger.info(log_message)
+            offset = 0
+            while offset >= 0:
+                bento_list = self.list(
+                    bento_name=bento_name,
+                    labels=labels,
+                    offset=offset,
+                    limit=delete_list_limit,
+                )
+                offset += delete_list_limit
+                # Stop the loop, when no more bentos
+                if len(bento_list) == 0:
+                    break
+                else:
+                    for bento in bento_list:
+                        self._delete_bento_bundle(
+                            f'{bento.name}:{bento.version}', require_confirm
+                        )
 
     def containerize(self, bento, tag=None, build_args=None, push=False):
         """
diff --git a/bentoml/yatai/repository/metadata_store.py b/bentoml/yatai/repository/metadata_store.py
index aac4e0ead79..ef6750fd518 100644
--- a/bentoml/yatai/repository/metadata_store.py
+++ b/bentoml/yatai/repository/metadata_store.py
@@ -251,7 +251,9 @@ def list(
                 # filter_by apply filtering criterion to a copy of the query
                 query = query.filter_by(name=bento_name)
             query = query.filter_by(deleted=False)
-            if label_selectors.match_labels or label_selectors.match_expressions:
+            if label_selectors is not None and (
+                label_selectors.match_labels or label_selectors.match_expressions
+            ):
                 bento_ids = filter_label_query(
                     sess, RESOURCE_TYPE.bento, label_selectors
                 )
diff --git a/docs/source/api/yatai_client.rst b/docs/source/api/yatai_client.rst
index fddce2c968f..96e33d0e85d 100644
--- a/docs/source/api/yatai_client.rst
+++ b/docs/source/api/yatai_client.rst
@@ -13,8 +13,6 @@ Yatai Client
 
     .. automethod:: bentoml.yatai.client.bento_repository_api.BentoRepositoryAPIClient.delete
 
-    .. automethod:: bentoml.yatai.client.bento_repository_api.BentoRepositoryAPIClient.prune
-
     .. automethod:: bentoml.yatai.client.bento_repository_api.BentoRepositoryAPIClient.containerize
 
 .. spelling::
diff --git a/tests/integration/yatai_server/test_delete.py b/tests/integration/yatai_server/test_delete.py
new file mode 100644
index 00000000000..7cb1208cabc
--- /dev/null
+++ b/tests/integration/yatai_server/test_delete.py
@@ -0,0 +1,119 @@
+import logging
+import uuid
+
+from click.testing import CliRunner
+
+from bentoml.cli import create_bentoml_cli
+from tests.bento_service_examples.iris_classifier import IrisClassifier
+from bentoml.yatai.client import get_yatai_client
+
+logger = logging.getLogger('bentoml.test')
+
+
+def test_delete_single_bento(bento_service):
+    yc = get_yatai_client()
+    # Remove all other bentos. Clean state
+    deleted_version = uuid.uuid4().hex[0:8]
+    yc.repository.delete(prune=True)
+
+    bento_service.save(version=deleted_version)
+    yc.repository.delete(
+        bento_name=bento_service.name, bento_version=deleted_version,
+    )
+    bentos = yc.repository.list()
+    assert len(bentos) == 0
+
+    another_deleted_version = uuid.uuid4().hex[0:10]
+    bento_service.save(version=another_deleted_version)
+    yc.repository.delete(f'{bento_service.name}:{bento_service.version}')
+    bentos = yc.repository.list()
+    assert len(bentos) == 0
+
+    # Clean up existing bentos
+    yc.repository.delete(prune=True)
+
+
+def test_delete_bentos_base_on_labels(bento_service):
+    yc = get_yatai_client()
+    yc.repository.delete(prune=True)
+    bento_service.save(version=uuid.uuid4().hex[0:8], labels={'cohort': '100'})
+    bento_service.save(version=uuid.uuid4().hex[0:8], labels={'cohort': '110'})
+    bento_service.save(version=uuid.uuid4().hex[0:8], labels={'cohort': '120'})
+
+    yc.repository.delete(labels='cohort in (100, 110)')
+    bentos = yc.repository.list()
+    assert len(bentos) == 1
+    # Clean up existing bentos
+    yc.repository.delete(prune=True)
+
+
+def test_delete_bentos_base_on_name(bento_service):
+    yc = get_yatai_client()
+    yc.repository.delete(prune=True)
+    bento_service.save(version=uuid.uuid4().hex[0:8])
+    bento_service.save(version=uuid.uuid4().hex[0:8])
+    iris = IrisClassifier()
+    iris.save()
+
+    yc.repository.delete(bento_name=bento_service.name)
+    bentos = yc.repository.list()
+    assert len(bentos) == 1
+    # Clean up existing bentos
+    yc.repository.delete(prune=True)
+
+
+def test_delete_bentos_on_name_and_labels(bento_service):
+    yc = get_yatai_client()
+    yc.repository.delete(prune=True)
+    bento_service.save(version=uuid.uuid4().hex[0:8], labels={'dataset': '20201212'})
+    bento_service.save(version=uuid.uuid4().hex[0:8], labels={'dataset': '20201212'})
+    bento_service.save(version=uuid.uuid4().hex[0:8], labels={'dataset': '20210101'})
+    iris = IrisClassifier()
+    iris.save()
+
+    yc.repository.delete(bento_name=bento_service.name, labels='dataset=20201212')
+    bentos = yc.repository.list()
+    assert len(bentos) == 2
+    # Clean up existing bentos
+    yc.repository.delete(prune=True)
+
+
+def test_delete_bentos_with_tags(bento_service):
+    yc = get_yatai_client()
+    yc.repository.delete(prune=True)
+    version_one = uuid.uuid4().hex[0:8]
+    version_two = uuid.uuid4().hex[0:8]
+    version_three = uuid.uuid4().hex[0:8]
+    bento_service.save(version=version_one)
+    bento_service.save(version=version_two)
+    bento_service.save(version=version_three)
+    yc.repository.delete(bento_tag=f'{bento_service.name}:{version_one}')
+    bentos = yc.repository.list()
+    assert len(bentos) == 2
+
+    runner = CliRunner()
+    cli = create_bentoml_cli()
+    print(cli.commands['delete'])
+    runner.invoke(
+        cli.commands['delete'],
+        [
+            '--tag',
+            f'{bento_service.name}:{version_two},{bento_service.name}:{version_three}',
+            '-y',
+        ],
+    )
+    bentos = yc.repository.list()
+    assert len(bentos) == 0
+    # Clean up existing bentos
+    yc.repository.delete(prune=True)
+
+
+def test_delete_all_bentos(bento_service):
+    bento_service.save(version=uuid.uuid4().hex[0:8])
+    bento_service.save(version=uuid.uuid4().hex[0:8])
+    bento_service.save(version=uuid.uuid4().hex[0:8])
+
+    yc = get_yatai_client()
+    yc.repository.delete(prune=True)
+    bentos = yc.repository.list()
+    assert len(bentos) == 0


Code review comments:
diff_hunk: @@ -175,39 +172,80 @@ def list_bentos(
         _print_bentos_info(result, output)
 
     @cli.command(
-        help='Delete bento. To delete multiple bentos provide the name '
-        'version tag separated by "," for example "bentoml delete name:v1,name:v2',
+        help='Delete bentos. To delete a bento service use "--bento-name" and '
+        '"--bento-version" option: "bentoml delete --bento-name IrisClassifier '
path: bentoml/cli/bento_management.py
text: @user1:
update the doc here to --name --version

@user1:
and add --tag

diff_hunk: @@ -175,39 +172,80 @@ def list_bentos(
         _print_bentos_info(result, output)
 
     @cli.command(
-        help='Delete bento. To delete multiple bentos provide the name '
-        'version tag separated by "," for example "bentoml delete name:v1,name:v2',
+        help='Delete bentos. To delete a bento service use "--bento-name" and '
+        '"--bento-version" option: "bentoml delete --bento-name IrisClassifier '
+        '--bento-version 0.1.0". To delete multiple bentos, use "--bento-name" and/or '
+        '"--labels" to filter. To delete all bento services use "--all" option'
+    )
+    @click.option(
+        '--all', is_flag=True, help='Use this flag to remove all BentoServices'
+    )
+    @click.option(
+        '--tag',
path: bentoml/cli/bento_management.py
text: @user1:
this will still be a breaking change right? since it used to be an argument?

@author:
Yes, this would be a breaking change.

@user1:
@author can you make it non-breaking and support the old format?