Repository: hyperledger/aries-cloudagent-python

Issue:
Update the logic for performing upgrades between releases
The following issues were identified in trying to upgrade a `0.8.0-rc0` deployment to `0.8.0`.  All the issues are relatively straight forward, so putting them all together into one issue.

### Issue: How to force an upgrade:
 
 - Decision is to have both `--force-upgrade` and `--from-version`
 
### Issue: What if the from-version is not in the `upgrade` yaml?  It would be nice to have it fallback to the one previous to the `--from-version` or in secure storage.
 
 - Decision: Make the code smarter and have it find the first version equal to or greater than the `from` version (from args or storage) and apply all the upgrades including and to the current version.  This becomes the literal from to use.
 - Further -- if there is more than one identical update -- run it only once. See the example above -- all of the updates are to `resave-ConnRec` so it really only needs to be done once.

### Issue: Once a version is in storage, cannot move to a newer version without running the `upgrade` command.

- Decision: Change to only stop if there is an upgrade required between versions -- and if not, update the storage version.

### Issue: On a new deployment, there is no version put into storage.

- Decision: Add to the create storage the additional of the version record.

Issue: No version in storage in existing deployments.

- Decision: Recommend that the deployment have an `upgrade --from-version...` command.

Issue: No warning currently produced when version in storage is not found.

- Decision: Add the warning, with details about what to do.

### Issue: Can we run an upgrade automatically when deploying a new version. Current behaviour is to abort the startup.

- Option: Auto-upgrade.
   - Risk: Multiple instances try to run the upgrade could cause unexpected side effects.
      - Same as the next problem -- must be run offline
      - ~~Need a semaphore or equivalent if the upgrade must be run once and only once.~~
   - Risk: Some updates MAY require running the upgrade offline.
      - Flag in YML file to say MUST be run as an explicit upgrade.

### Issue: The current expectation is that in the YML, the version for the change is the "from" version -- e.g. when upgrading from a designated release.

- Decision: The version associated with an upgrade command should be the version in when the upgrade should be run. E.g. Any version being upgraded to that version must run the upgrade steps.  So:
   - If the entry is `v0.8.0: Update_Connections` -- that command should be run from every pre-0.8.0 version when upgrade to 0.8.0 or beyond.  Further, if going from 0.7.5 to 0.9.0, they would run the `0.8.0` upgrade.



Discussion:
@shaangill025 -- if you think any of these are "too big" for what is an urgent update, please feel free to propose we separate out the change into a separate issue to be done in a later release.  We can decide as a group how urgent it is and how therefore we will handle it.

Thanks

Resolving patch:
diff --git a/aries_cloudagent/commands/upgrade.py b/aries_cloudagent/commands/upgrade.py
--- a/aries_cloudagent/commands/upgrade.py
+++ b/aries_cloudagent/commands/upgrade.py
@@ -1,11 +1,12 @@
 """Upgrade command for handling breaking changes when updating ACA-PY versions."""
 
 import asyncio
+import logging
 import yaml
 
 from configargparse import ArgumentParser
 from packaging import version as package_version
-from typing import Callable, Sequence, Optional
+from typing import Callable, Sequence, Optional, List
 
 from ..core.profile import Profile
 from ..config import argparse as arg
@@ -25,6 +26,7 @@
 DEFAULT_UPGRADE_CONFIG_PATH = (
     "./aries_cloudagent/commands/default_version_upgrade_config.yml"
 )
+LOGGER = logging.getLogger(__name__)
 
 
 class UpgradeError(BaseError):
@@ -36,23 +38,13 @@ class VersionUpgradeConfig:
 
     def __init__(self, config_path: str = None):
         """Initialize config for use during upgrade process."""
-        self.function_map_config = {}
+        self.function_map_config = UPGRADE_EXISTING_RECORDS_FUNCTION_MAPPING
         self.upgrade_configs = {}
-        self.setup_executable_map_config(CONFIG_v7_3)
         if config_path:
             self.setup_version_upgrade_config(config_path)
         else:
             self.setup_version_upgrade_config(DEFAULT_UPGRADE_CONFIG_PATH)
 
-    def setup_executable_map_config(self, config_dict: dict):
-        """Set ups config with reference to functions mapped to versions."""
-        for version, config in config_dict.items():
-            self.function_map_config[version] = {}
-            if "update_existing_function_inst" in config:
-                self.function_map_config[version][
-                    "update_existing_function_inst"
-                ] = config.get("update_existing_function_inst")
-
     def setup_version_upgrade_config(self, path: str):
         """Set ups config dict from the provided YML file."""
         with open(path, "r") as stream:
@@ -73,19 +65,23 @@ def setup_version_upgrade_config(self, path: str):
                             "resave_records"
                         ).get("base_exch_record_path")
                 version_config_dict[version]["resave_records"] = recs_list
-                version_config_dict[version]["update_existing_records"] = (
-                    provided_config.get("update_existing_records") or False
-                )
+                config_key_set = set(provided_config.keys())
+                try:
+                    config_key_set.remove("resave_records")
+                except KeyError:
+                    pass
+                for executable in config_key_set:
+                    version_config_dict[version][executable] = (
+                        provided_config.get(executable) or False
+                    )
             if version_config_dict == {}:
                 raise UpgradeError(f"No version configs found in {path}")
             self.upgrade_configs = version_config_dict
 
-    def get_update_existing_func(self, ver: str) -> Optional[Callable]:
-        """Return callable update_existing_records function for specific version."""
-        if ver in self.function_map_config:
-            return self.function_map_config.get(ver).get(
-                "update_existing_function_inst"
-            )
+    def get_callable(self, executable: str) -> Optional[Callable]:
+        """Return callable function for executable name."""
+        if executable in self.function_map_config:
+            return self.function_map_config.get(executable)
         else:
             return None
 
@@ -95,6 +91,49 @@ def init_argument_parser(parser: ArgumentParser):
     return arg.load_argument_groups(parser, *arg.group.get_registered(arg.CAT_UPGRADE))
 
 
+def get_upgrade_version_list(
+    from_version: str,
+    sorted_version_list: Optional[List] = None,
+    config_path: Optional[str] = None,
+) -> List:
+    """Get available versions from the upgrade config."""
+    if not sorted_version_list:
+        version_upgrade_config_inst = VersionUpgradeConfig(config_path)
+        upgrade_configs = version_upgrade_config_inst.upgrade_configs
+        versions_found_in_config = upgrade_configs.keys()
+        sorted_version_list = sorted(
+            versions_found_in_config, key=lambda x: package_version.parse(x)
+        )
+
+    version_list = []
+    for version in sorted_version_list:
+        if package_version.parse(version) >= package_version.parse(from_version):
+            version_list.append(version)
+    return version_list
+
+
+async def add_version_record(profile: Profile, version: str):
+    """Add an acapy_version storage record for provided version."""
+    async with profile.session() as session:
+        storage = session.inject(BaseStorage)
+        try:
+            version_storage_record = await storage.find_record(
+                type_filter=RECORD_TYPE_ACAPY_VERSION, tag_query={}
+            )
+        except StorageNotFoundError:
+            version_storage_record = None
+        if not version_storage_record:
+            await storage.add_record(
+                StorageRecord(
+                    RECORD_TYPE_ACAPY_VERSION,
+                    version,
+                )
+            )
+        else:
+            await storage.update_record(version_storage_record, version, {})
+    LOGGER.info(f"{RECORD_TYPE_ACAPY_VERSION} storage record set to {version}")
+
+
 async def upgrade(settings: dict):
     """Perform upgradation steps."""
     context_builder = DefaultContextBuilder(settings)
@@ -104,114 +143,147 @@ async def upgrade(settings: dict):
             settings.get("upgrade.config_path")
         )
         upgrade_configs = version_upgrade_config_inst.upgrade_configs
-        root_profile, public_did = await wallet_config(context)
-        version_storage_record = None
+        root_profile, _ = await wallet_config(context)
         upgrade_to_version = f"v{__version__}"
         versions_found_in_config = upgrade_configs.keys()
         sorted_versions_found_in_config = sorted(
             versions_found_in_config, key=lambda x: package_version.parse(x)
         )
+        upgrade_from_version_config = None
+        upgrade_from_version_setting = None
+        upgrade_from_version = None
         async with root_profile.session() as session:
             storage = session.inject(BaseStorage)
             try:
                 version_storage_record = await storage.find_record(
                     type_filter=RECORD_TYPE_ACAPY_VERSION, tag_query={}
                 )
-                upgrade_from_version = version_storage_record.value
-                if "upgrade.from_version" in settings:
-                    print(
-                        (
-                            f"version {upgrade_from_version} found in storage"
-                            ", --from-version will be ignored."
-                        )
-                    )
+                upgrade_from_version_config = version_storage_record.value
             except StorageNotFoundError:
-                if "upgrade.from_version" in settings:
-                    upgrade_from_version = settings.get("upgrade.from_version")
-                else:
-                    upgrade_from_version = sorted_versions_found_in_config[-1]
-                    print(
-                        "No ACA-Py version found in wallet storage and "
-                        "no --from-version specified. Selecting "
-                        f"{upgrade_from_version} as --from-version from "
-                        "the config."
+                LOGGER.info("No ACA-Py version found in wallet storage.")
+                version_storage_record = None
+
+            if "upgrade.from_version" in settings:
+                upgrade_from_version_setting = settings.get("upgrade.from_version")
+                LOGGER.info(
+                    (
+                        f"Selecting {upgrade_from_version_setting} as "
+                        "--from-version from the config."
                     )
-        if upgrade_from_version == upgrade_to_version:
-            print(
-                f"Version {upgrade_from_version} to upgrade from and "
-                f"current version to upgrade to {upgrade_to_version} "
-                "are same."
+                )
+
+        force_upgrade_flag = settings.get("upgrade.force_upgrade") or False
+        if upgrade_from_version_config and upgrade_from_version_setting:
+            if (
+                package_version.parse(upgrade_from_version_config)
+                > package_version.parse(upgrade_from_version_setting)
+            ) and force_upgrade_flag:
+                upgrade_from_version = upgrade_from_version_setting
+            else:
+                upgrade_from_version = upgrade_from_version_config
+        if (
+            not upgrade_from_version
+            and not upgrade_from_version_config
+            and upgrade_from_version_setting
+        ):
+            upgrade_from_version = upgrade_from_version_setting
+        if (
+            not upgrade_from_version
+            and upgrade_from_version_config
+            and not upgrade_from_version_setting
+        ):
+            upgrade_from_version = upgrade_from_version_config
+        if not upgrade_from_version:
+            raise UpgradeError(
+                "No upgrade from version found in wallet or settings [--from-version]"
             )
-        else:
-            if upgrade_from_version not in sorted_versions_found_in_config:
-                raise UpgradeError(
-                    f"No upgrade configuration found for {upgrade_from_version}"
+        upgrade_version_in_config = get_upgrade_version_list(
+            sorted_version_list=sorted_versions_found_in_config,
+            from_version=upgrade_from_version,
+        )
+        to_update_flag = False
+        if upgrade_from_version == upgrade_to_version:
+            LOGGER.info(
+                (
+                    f"Version {upgrade_from_version} to upgrade from and "
+                    f"current version to upgrade to {upgrade_to_version} "
+                    "are same. You can apply upgrade from a lower "
+                    "version by running the upgrade command with "
+                    f"--from-version [< {upgrade_to_version}] and "
+                    "--force-upgrade"
                 )
-            upgrade_from_version_index = sorted_versions_found_in_config.index(
-                upgrade_from_version
             )
-            for config_from_version in sorted_versions_found_in_config[
-                upgrade_from_version_index:
-            ]:
-                print(f"Running upgrade process for {config_from_version}")
+        else:
+            resave_record_path_sets = set()
+            executables_call_set = set()
+            for config_from_version in upgrade_version_in_config:
+                LOGGER.info(f"Running upgrade process for {config_from_version}")
                 upgrade_config = upgrade_configs.get(config_from_version)
                 # Step 1 re-saving all BaseRecord and BaseExchangeRecord
                 if "resave_records" in upgrade_config:
                     resave_record_paths = upgrade_config.get("resave_records")
                     for record_path in resave_record_paths:
-                        try:
-                            rec_type = ClassLoader.load_class(record_path)
-                        except ClassNotFoundError as err:
-                            raise UpgradeError(
-                                f"Unknown Record type {record_path}"
-                            ) from err
-                        if not issubclass(rec_type, BaseRecord):
-                            raise UpgradeError(
-                                f"Only BaseRecord can be resaved, found: {str(rec_type)}"
-                            )
-                        async with root_profile.session() as session:
-                            all_records = await rec_type.query(session)
-                            for record in all_records:
-                                await record.save(
-                                    session,
-                                    reason="re-saving record during the upgrade process",
-                                )
-                            if len(all_records) == 0:
-                                print(f"No records of {str(rec_type)} found")
-                            else:
-                                print(
-                                    f"All recs of {str(rec_type)} successfully re-saved"
-                                )
+                        resave_record_path_sets.add(record_path)
+
                 # Step 2 Update existing records, if required
-                if (
-                    "update_existing_records" in upgrade_config
-                    and upgrade_config.get("update_existing_records") is True
-                ):
-                    update_existing_recs_callable = (
-                        version_upgrade_config_inst.get_update_existing_func(
-                            config_from_version
-                        )
+                config_key_set = set(upgrade_config.keys())
+                try:
+                    config_key_set.remove("resave_records")
+                except KeyError:
+                    pass
+                for callable_name in list(config_key_set):
+                    if upgrade_config.get(callable_name) is False:
+                        continue
+                    executables_call_set.add(callable_name)
+
+            if len(resave_record_path_sets) >= 1 or len(executables_call_set) >= 1:
+                to_update_flag = True
+            for record_path in resave_record_path_sets:
+                try:
+                    rec_type = ClassLoader.load_class(record_path)
+                except ClassNotFoundError as err:
+                    raise UpgradeError(f"Unknown Record type {record_path}") from err
+                if not issubclass(rec_type, BaseRecord):
+                    raise UpgradeError(
+                        f"Only BaseRecord can be resaved, found: {str(rec_type)}"
                     )
-                    if not update_existing_recs_callable:
-                        raise UpgradeError(
-                            "No update_existing_records function "
-                            f"specified for {config_from_version}"
+                async with root_profile.session() as session:
+                    all_records = await rec_type.query(session)
+                    for record in all_records:
+                        await record.save(
+                            session,
+                            reason="re-saving record during the upgrade process",
+                        )
+                    if len(all_records) == 0:
+                        LOGGER.info(f"No records of {str(rec_type)} found")
+                    else:
+                        LOGGER.info(
+                            f"All recs of {str(rec_type)} successfully re-saved"
                         )
-                    await update_existing_recs_callable(root_profile)
+            for callable_name in executables_call_set:
+                _callable = version_upgrade_config_inst.get_callable(callable_name)
+                if not _callable:
+                    raise UpgradeError(f"No function specified for {callable_name}")
+                await _callable(root_profile)
 
         # Update storage version
-        async with root_profile.session() as session:
-            storage = session.inject(BaseStorage)
-            if not version_storage_record:
-                await storage.add_record(
-                    StorageRecord(
-                        RECORD_TYPE_ACAPY_VERSION,
-                        upgrade_to_version,
+        if to_update_flag:
+            async with root_profile.session() as session:
+                storage = session.inject(BaseStorage)
+                if not version_storage_record:
+                    await storage.add_record(
+                        StorageRecord(
+                            RECORD_TYPE_ACAPY_VERSION,
+                            upgrade_to_version,
+                        )
                     )
-                )
-            else:
-                await storage.update_record(
-                    version_storage_record, upgrade_to_version, {}
+                else:
+                    await storage.update_record(
+                        version_storage_record, upgrade_to_version, {}
+                    )
+                LOGGER.info(
+                    f"{RECORD_TYPE_ACAPY_VERSION} storage record "
+                    f"set to {upgrade_to_version}"
                 )
         await root_profile.close()
     except BaseError as e:
@@ -247,12 +319,8 @@ def main():
         execute()
 
 
-# Update every release
-CONFIG_v7_3 = {
-    "v0.7.2": {
-        "update_existing_function_inst": update_existing_records,
-    },
+UPGRADE_EXISTING_RECORDS_FUNCTION_MAPPING = {
+    "update_existing_records": update_existing_records
 }
 
-
 main()
diff --git a/aries_cloudagent/config/argparse.py b/aries_cloudagent/config/argparse.py
--- a/aries_cloudagent/config/argparse.py
+++ b/aries_cloudagent/config/argparse.py
@@ -2053,6 +2053,17 @@ def add_arguments(self, parser: ArgumentParser):
             ),
         )
 
+        parser.add_argument(
+            "--force-upgrade",
+            action="store_true",
+            env_var="ACAPY_UPGRADE_FORCE_UPGRADE",
+            help=(
+                "Forces the '—from-version' argument to override the version "
+                "retrieved from secure storage when calculating upgrades to "
+                "be run."
+            ),
+        )
+
     def get_settings(self, args: Namespace) -> dict:
         """Extract ACA-Py upgrade process settings."""
         settings = {}
@@ -2060,4 +2071,6 @@ def get_settings(self, args: Namespace) -> dict:
             settings["upgrade.config_path"] = args.upgrade_config_path
         if args.from_version:
             settings["upgrade.from_version"] = args.from_version
+        if args.force_upgrade:
+            settings["upgrade.force_upgrade"] = args.force_upgrade
         return settings
diff --git a/aries_cloudagent/core/conductor.py b/aries_cloudagent/core/conductor.py
--- a/aries_cloudagent/core/conductor.py
+++ b/aries_cloudagent/core/conductor.py
@@ -26,6 +26,11 @@
 from ..config.logging import LoggingConfigurator
 from ..config.provider import ClassProvider
 from ..config.wallet import wallet_config
+from ..commands.upgrade import (
+    get_upgrade_version_list,
+    add_version_record,
+    upgrade,
+)
 from ..core.profile import Profile
 from ..indy.verifier import IndyVerifier
 
@@ -311,26 +316,43 @@ async def start(self) -> None:
         )
 
         # record ACA-Py version in Wallet, if needed
+        from_version = None
+        agent_version = f"v{__version__}"
         async with self.root_profile.session() as session:
             storage: BaseStorage = session.context.inject(BaseStorage)
-            agent_version = f"v{__version__}"
             try:
                 record = await storage.find_record(
                     type_filter=RECORD_TYPE_ACAPY_VERSION,
                     tag_query={},
                 )
-                if record.value != agent_version:
-                    LOGGER.exception(
-                        (
-                            f"Wallet storage version {record.value} "
-                            "does not match this ACA-Py agent "
-                            f"version {agent_version}. Run aca-py "
-                            "upgrade command to fix this."
-                        )
-                    )
-                    raise
+                from_version = record.value
             except StorageNotFoundError:
-                pass
+                LOGGER.warning(("Wallet version storage record not found."))
+        from_version = from_version or self.root_profile.settings.get(
+            "upgrade.config_path"
+        )
+        if from_version:
+            config_available_list = get_upgrade_version_list(
+                config_path=self.root_profile.settings.get("upgrade.config_path"),
+                from_version=from_version,
+            )
+            if len(config_available_list) >= 1 and (
+                from_version != agent_version
+                or self.root_profile.settings.get("upgrade.force_upgrade")
+            ):
+                await upgrade(self.root_profile.settings)
+        else:
+            LOGGER.warning(
+                (
+                    "No upgrade from version was found from wallet or via"
+                    " --from-version startup argument. "
+                    f"{RECORD_TYPE_ACAPY_VERSION} storage record will be "
+                    f"set to {agent_version}. You can run run the upgrade "
+                    "command with --from-version and --force-upgrade to "
+                    "upgrade later."
+                )
+            )
+            await add_version_record(profile=self.root_profile, version=agent_version)
 
         # Create a static connection for use by the test-suite
         if context.settings.get("debug.test_suite_endpoint"):


Test patch:
diff --git a/aries_cloudagent/commands/tests/test_upgrade.py b/aries_cloudagent/commands/tests/test_upgrade.py
--- a/aries_cloudagent/commands/tests/test_upgrade.py
+++ b/aries_cloudagent/commands/tests/test_upgrade.py
@@ -17,10 +17,7 @@ class TestUpgrade(AsyncTestCase):
     async def setUp(self):
         self.session = InMemoryProfile.test_session()
         self.profile = self.session.profile
-
-        self.session_storage = InMemoryProfile.test_session()
-        self.profile_storage = self.session_storage.profile
-        self.storage = self.session_storage.inject(BaseStorage)
+        self.storage = self.session.inject(BaseStorage)
         record = StorageRecord(
             "acapy_version",
             "v0.7.2",
@@ -37,7 +34,7 @@ async def test_upgrade_storage_from_version_included(self):
             "wallet_config",
             async_mock.CoroutineMock(
                 return_value=(
-                    self.profile_storage,
+                    self.profile,
                     async_mock.CoroutineMock(did="public DID", verkey="verkey"),
                 )
             ),
@@ -61,7 +58,7 @@ async def test_upgrade_storage_missing_from_version(self):
             "wallet_config",
             async_mock.CoroutineMock(
                 return_value=(
-                    self.profile_storage,
+                    self.profile,
                     async_mock.CoroutineMock(did="public DID", verkey="verkey"),
                 )
             ),
@@ -98,6 +95,10 @@ async def test_upgrade_from_version(self):
             )
 
     async def test_upgrade_callable(self):
+        version_storage_record = await self.storage.find_record(
+            type_filter="acapy_version", tag_query={}
+        )
+        await self.storage.delete_record(version_storage_record)
         with async_mock.patch.object(
             test_module,
             "wallet_config",
@@ -139,7 +140,7 @@ async def test_upgrade_x_same_version(self):
             "wallet_config",
             async_mock.CoroutineMock(
                 return_value=(
-                    self.profile_storage,
+                    self.profile,
                     async_mock.CoroutineMock(did="public DID", verkey="verkey"),
                 )
             ),
@@ -151,6 +152,10 @@ async def test_upgrade_x_same_version(self):
             )
 
     async def test_upgrade_missing_from_version(self):
+        version_storage_record = await self.storage.find_record(
+            type_filter="acapy_version", tag_query={}
+        )
+        await self.storage.delete_record(version_storage_record)
         with async_mock.patch.object(
             test_module,
             "wallet_config",
@@ -167,13 +172,19 @@ async def test_upgrade_missing_from_version(self):
         ), async_mock.patch.object(
             ConnRecord, "save", async_mock.CoroutineMock()
         ):
-            await test_module.upgrade(
-                {
-                    "upgrade.config_path": "./aries_cloudagent/commands/default_version_upgrade_config.yml",
-                }
-            )
+            with self.assertRaises(UpgradeError) as ctx:
+                await test_module.upgrade(
+                    {
+                        "upgrade.config_path": "./aries_cloudagent/commands/default_version_upgrade_config.yml",
+                    }
+                )
+            assert "No upgrade from version found in wallet or" in str(ctx.exception)
 
     async def test_upgrade_x_callable_not_set(self):
+        version_storage_record = await self.storage.find_record(
+            type_filter="acapy_version", tag_query={}
+        )
+        await self.storage.delete_record(version_storage_record)
         with async_mock.patch.object(
             test_module,
             "wallet_config",
@@ -196,7 +207,7 @@ async def test_upgrade_x_callable_not_set(self):
                         },
                         "update_existing_records": True,
                     },
-                    "v0.6.0": {"update_existing_records": True},
+                    "v0.6.0": {"update_existing_records_b": True},
                 }
             ),
         ):
@@ -206,7 +217,7 @@ async def test_upgrade_x_callable_not_set(self):
                         "upgrade.from_version": "v0.6.0",
                     }
                 )
-            assert "No update_existing_records function specified" in str(ctx.exception)
+            assert "No function specified for" in str(ctx.exception)
 
     async def test_upgrade_x_class_not_found(self):
         with async_mock.patch.object(
@@ -268,7 +279,8 @@ async def test_execute(self):
                     "--upgrade-config",
                     "./aries_cloudagent/config/tests/test-acapy-upgrade-config.yaml",
                     "--from-version",
-                    "v0.7.2",
+                    "v0.7.0",
+                    "--force-upgrade",
                 ]
             )
 
@@ -305,17 +317,7 @@ async def test_upgrade_x_invalid_record_type(self):
                 )
             assert "Only BaseRecord can be resaved" in str(ctx.exception)
 
-    async def test_upgrade_x_invalid_config(self):
-        with async_mock.patch.object(
-            test_module.yaml,
-            "safe_load",
-            async_mock.MagicMock(return_value={}),
-        ):
-            with self.assertRaises(UpgradeError) as ctx:
-                await test_module.upgrade({})
-            assert "No version configs found in" in str(ctx.exception)
-
-    async def test_upgrade_x_from_version_not_in_config(self):
+    async def test_upgrade_force(self):
         with async_mock.patch.object(
             test_module,
             "wallet_config",
@@ -325,14 +327,62 @@ async def test_upgrade_x_from_version_not_in_config(self):
                     async_mock.CoroutineMock(did="public DID", verkey="verkey"),
                 )
             ),
+        ), async_mock.patch.object(
+            test_module.yaml,
+            "safe_load",
+            async_mock.MagicMock(
+                return_value={
+                    "v0.7.2": {
+                        "resave_records": {
+                            "base_record_path": [
+                                "aries_cloudagent.connections.models.conn_record.ConnRecord"
+                            ],
+                        },
+                        "update_existing_records": True,
+                    },
+                    "v0.7.3": {
+                        "update_existing_records": True,
+                    },
+                    "v0.7.1": {
+                        "update_existing_records": False,
+                    },
+                }
+            ),
+        ):
+            await test_module.upgrade(
+                {
+                    "upgrade.from_version": "v0.7.0",
+                    "upgrade.force_upgrade": True,
+                }
+            )
+
+    async def test_get_upgrade_version_list(self):
+        assert len(test_module.get_upgrade_version_list(from_version="v0.7.2")) >= 1
+
+    async def test_add_version_record(self):
+        await test_module.add_version_record(self.profile, "v0.7.4")
+        version_storage_record = await self.storage.find_record(
+            type_filter="acapy_version", tag_query={}
+        )
+        assert version_storage_record.value == "v0.7.4"
+        await self.storage.delete_record(version_storage_record)
+        with self.assertRaises(test_module.StorageNotFoundError):
+            await self.storage.find_record(type_filter="acapy_version", tag_query={})
+        await test_module.add_version_record(self.profile, "v0.7.5")
+        version_storage_record = await self.storage.find_record(
+            type_filter="acapy_version", tag_query={}
+        )
+        assert version_storage_record.value == "v0.7.5"
+
+    async def test_upgrade_x_invalid_config(self):
+        with async_mock.patch.object(
+            test_module.yaml,
+            "safe_load",
+            async_mock.MagicMock(return_value={}),
         ):
             with self.assertRaises(UpgradeError) as ctx:
-                await test_module.upgrade(
-                    {
-                        "upgrade.from_version": "v1.2.3",
-                    }
-                )
-            assert "No upgrade configuration found for" in str(ctx.exception)
+                await test_module.upgrade({})
+            assert "No version configs found in" in str(ctx.exception)
 
     def test_main(self):
         with async_mock.patch.object(
diff --git a/aries_cloudagent/config/tests/test_argparse.py b/aries_cloudagent/config/tests/test_argparse.py
--- a/aries_cloudagent/config/tests/test_argparse.py
+++ b/aries_cloudagent/config/tests/test_argparse.py
@@ -111,6 +111,7 @@ async def test_upgrade_config(self):
                 "./aries_cloudagent/config/tests/test-acapy-upgrade-config.yml",
                 "--from-version",
                 "v0.7.2",
+                "--force-upgrade",
             ]
         )
 
@@ -118,6 +119,7 @@ async def test_upgrade_config(self):
             result.upgrade_config_path
             == "./aries_cloudagent/config/tests/test-acapy-upgrade-config.yml"
         )
+        assert result.force_upgrade is True
 
         settings = group.get_settings(result)
 
diff --git a/aries_cloudagent/core/tests/test_conductor.py b/aries_cloudagent/core/tests/test_conductor.py
--- a/aries_cloudagent/core/tests/test_conductor.py
+++ b/aries_cloudagent/core/tests/test_conductor.py
@@ -101,7 +101,7 @@ async def build_context(self) -> InjectionContext:
 
 
 class TestConductor(IsolatedAsyncioTestCase, Config, TestDIDs):
-    async def test_startup(self):
+    async def test_startup_version_record_exists(self):
         builder: ContextBuilder = StubContextBuilder(self.test_settings)
         conductor = test_module.Conductor(builder)
 
@@ -114,9 +114,67 @@ async def test_startup(self):
         ) as mock_logger, async_mock.patch.object(
             BaseStorage,
             "find_record",
-            async_mock.AsyncMock(
-                return_value=async_mock.MagicMock(value=f"v{__version__}")
+            async_mock.AsyncMock(return_value=async_mock.MagicMock(value="v0.7.3")),
+        ), async_mock.patch.object(
+            test_module,
+            "get_upgrade_version_list",
+            async_mock.MagicMock(
+                return_value=["v0.7.4", "0.7.5", "v0.8.0rc1", "v8.0.0"]
             ),
+        ), async_mock.patch.object(
+            test_module,
+            "upgrade",
+            async_mock.AsyncMock(),
+        ):
+            mock_outbound_mgr.return_value.registered_transports = {
+                "test": async_mock.MagicMock(schemes=["http"])
+            }
+            await conductor.setup()
+
+            session = await conductor.root_profile.session()
+
+            wallet = session.inject(BaseWallet)
+            await wallet.create_public_did(
+                SOV,
+                ED25519,
+            )
+
+            mock_inbound_mgr.return_value.setup.assert_awaited_once()
+            mock_outbound_mgr.return_value.setup.assert_awaited_once()
+
+            mock_inbound_mgr.return_value.registered_transports = {}
+            mock_outbound_mgr.return_value.registered_transports = {}
+
+            await conductor.start()
+
+            mock_inbound_mgr.return_value.start.assert_awaited_once_with()
+            mock_outbound_mgr.return_value.start.assert_awaited_once_with()
+
+            mock_logger.print_banner.assert_called_once()
+
+            await conductor.stop()
+
+            mock_inbound_mgr.return_value.stop.assert_awaited_once_with()
+            mock_outbound_mgr.return_value.stop.assert_awaited_once_with()
+
+    async def test_startup_version_record_not_exists(self):
+        builder: ContextBuilder = StubContextBuilder(self.test_settings)
+        conductor = test_module.Conductor(builder)
+
+        with async_mock.patch.object(
+            test_module, "InboundTransportManager", autospec=True
+        ) as mock_inbound_mgr, async_mock.patch.object(
+            test_module, "OutboundTransportManager", autospec=True
+        ) as mock_outbound_mgr, async_mock.patch.object(
+            test_module, "LoggingConfigurator", autospec=True
+        ) as mock_logger, async_mock.patch.object(
+            BaseStorage,
+            "find_record",
+            async_mock.AsyncMock(side_effect=StorageNotFoundError()),
+        ), async_mock.patch.object(
+            test_module,
+            "add_version_record",
+            async_mock.AsyncMock(),
         ):
             mock_outbound_mgr.return_value.registered_transports = {
                 "test": async_mock.MagicMock(schemes=["http"])
@@ -1370,43 +1428,6 @@ async def test_setup_ledger_only_base(self):
             await conductor.setup()
             mock_genesis_load.assert_called_once()
 
-    async def test_startup_x_version_mismatch(self):
-        builder: ContextBuilder = StubContextBuilder(self.test_settings)
-        conductor = test_module.Conductor(builder)
-
-        with async_mock.patch.object(
-            test_module, "InboundTransportManager", autospec=True
-        ) as mock_inbound_mgr, async_mock.patch.object(
-            test_module, "OutboundTransportManager", autospec=True
-        ) as mock_outbound_mgr, async_mock.patch.object(
-            test_module, "LOGGER"
-        ) as mock_logger, async_mock.patch.object(
-            BaseStorage,
-            "find_record",
-            async_mock.AsyncMock(return_value=async_mock.MagicMock(value=f"v0.6.0")),
-        ):
-            mock_outbound_mgr.return_value.registered_transports = {
-                "test": async_mock.MagicMock(schemes=["http"])
-            }
-            await conductor.setup()
-
-            session = await conductor.root_profile.session()
-
-            wallet = session.inject(BaseWallet)
-            await wallet.create_public_did(
-                SOV,
-                ED25519,
-            )
-
-            mock_inbound_mgr.return_value.setup.assert_awaited_once()
-            mock_outbound_mgr.return_value.setup.assert_awaited_once()
-
-            mock_inbound_mgr.return_value.registered_transports = {}
-            mock_outbound_mgr.return_value.registered_transports = {}
-            with self.assertRaises(RuntimeError):
-                await conductor.start()
-                mock_logger.exception.assert_called_once()
-
     async def test_startup_x_no_storage_version(self):
         builder: ContextBuilder = StubContextBuilder(self.test_settings)
         conductor = test_module.Conductor(builder)
@@ -1421,6 +1442,10 @@ async def test_startup_x_no_storage_version(self):
             BaseStorage,
             "find_record",
             async_mock.AsyncMock(side_effect=StorageNotFoundError()),
+        ), async_mock.patch.object(
+            test_module,
+            "add_version_record",
+            async_mock.AsyncMock(),
         ):
             mock_outbound_mgr.return_value.registered_transports = {
                 "test": async_mock.MagicMock(schemes=["http"])