Repository: Azure/azure-cli

Issue:
Proposal: az account refresh
Sometimes I have new subscriptions associated to my account that did not exist when I did `azure login` so I have to do `azure login` again and go through the interactive login flow to get the new subscriptions I have show up on my local CLI.

Since CLI already has my working credentials, perhaps a new `az account refresh` can re-download my subscriptions list?


Discussion:
codecov-io: # [Codecov](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=h1) Report
> Merging [#4091](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=desc) into [master](https://codecov.io/gh/Azure/azure-cli/commit/b3c613ff92ce7bfc40caed5c86fb508c7e7d3710?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `79.03%`.

[![Impacted file tree graph](https://codecov.io/gh/Azure/azure-cli/pull/4091/graphs/tree.svg?token=2pog0TKvF8&src=pr&height=150&width=650)](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4091      +/-   ##
==========================================
+ Coverage   70.49%   70.49%   +<.01%     
==========================================
  Files         437      437              
  Lines       28086    28132      +46     
  Branches     4300     4314      +14     
==========================================
+ Hits        19800    19833      +33     
- Misses       7004     7017      +13     
  Partials     1282     1282
```


| [Impacted Files](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ofile/azure/cli/command\_modules/profile/\_params.py](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=tree#diff-c3JjL2NvbW1hbmRfbW9kdWxlcy9henVyZS1jbGktcHJvZmlsZS9henVyZS9jbGkvY29tbWFuZF9tb2R1bGVzL3Byb2ZpbGUvX3BhcmFtcy5weQ==) | `68.42% <100%> (+1.75%)` | :arrow_up: |
| [...rofile/azure/cli/command\_modules/profile/custom.py](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=tree#diff-c3JjL2NvbW1hbmRfbW9kdWxlcy9henVyZS1jbGktcHJvZmlsZS9henVyZS9jbGkvY29tbWFuZF9tb2R1bGVzL3Byb2ZpbGUvY3VzdG9tLnB5) | `37.07% <60%> (-0.86%)` | :arrow_down: |
| [src/azure-cli-core/azure/cli/core/\_profile.py](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=tree#diff-c3JjL2F6dXJlLWNsaS1jb3JlL2F6dXJlL2NsaS9jb3JlL19wcm9maWxlLnB5) | `86.44% <80.35%> (-0.96%)` | :arrow_down: |
| [...li-cloud/azure/cli/command\_modules/cloud/custom.py](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=tree#diff-c3JjL2NvbW1hbmRfbW9kdWxlcy9henVyZS1jbGktY2xvdWQvYXp1cmUvY2xpL2NvbW1hbmRfbW9kdWxlcy9jbG91ZC9jdXN0b20ucHk=) | `13.79% <0%> (ø)` | :arrow_up: |
| [...nent/azure/cli/command\_modules/component/custom.py](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=tree#diff-c3JjL2NvbW1hbmRfbW9kdWxlcy9henVyZS1jbGktY29tcG9uZW50L2F6dXJlL2NsaS9jb21tYW5kX21vZHVsZXMvY29tcG9uZW50L2N1c3RvbS5weQ==) | `16.23% <0%> (ø)` | :arrow_up: |
| [...dback/azure/cli/command\_modules/feedback/custom.py](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=tree#diff-c3JjL2NvbW1hbmRfbW9kdWxlcy9henVyZS1jbGktZmVlZGJhY2svYXp1cmUvY2xpL2NvbW1hbmRfbW9kdWxlcy9mZWVkYmFjay9jdXN0b20ucHk=) | `31.25% <0%> (ø)` | :arrow_up: |
| [src/azure-cli-core/azure/cli/core/util.py](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=tree#diff-c3JjL2F6dXJlLWNsaS1jb3JlL2F6dXJlL2NsaS9jb3JlL3V0aWwucHk=) | `70.66% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=footer). Last update [b3c613f...8ef9ee8](https://codecov.io/gh/Azure/azure-cli/pull/4091?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/src/azure-cli-core/azure/cli/core/_profile.py b/src/azure-cli-core/azure/cli/core/_profile.py
index 1d12fd835..7200ab600 100644
--- a/src/azure-cli-core/azure/cli/core/_profile.py
+++ b/src/azure-cli-core/azure/cli/core/_profile.py
@@ -57,6 +57,7 @@ _CLIENT_ID = '04b07795-8ddb-461a-bbee-02f9e1bf7b46'
 _COMMON_TENANT = 'common'
 
 _MSI_ACCOUNT_NAME = 'MSI'
+_TENANT_LEVEL_ACCOUNT_NAME = 'N/A(tenant level account)'
 
 
 def _authentication_context_factory(tenant, cache):
@@ -235,7 +236,7 @@ class Profile(object):
             s.id = '/subscriptions/' + t
             s.subscription = t
             s.tenant_id = t
-            s.display_name = 'N/A(tenant level account)'
+            s.display_name = _TENANT_LEVEL_ACCOUNT_NAME
             result.append(s)
         return result
 
@@ -263,7 +264,7 @@ class Profile(object):
         # use deepcopy as we don't want to persist these changes to file.
         return deepcopy(consolidated)
 
-    def _set_subscriptions(self, new_subscriptions):
+    def _set_subscriptions(self, new_subscriptions, merge=True):
         existing_ones = self.load_cached_subscriptions(all_clouds=True)
         active_one = next((x for x in existing_ones if x.get(_IS_DEFAULT_SUBSCRIPTION)), None)
         active_subscription_id = active_one[_SUBSCRIPTION_ID] if active_one else None
@@ -271,27 +272,31 @@ class Profile(object):
         default_sub_id = None
 
         # merge with existing ones
-        dic = collections.OrderedDict((x[_SUBSCRIPTION_ID], x) for x in existing_ones)
+        if merge:
+            dic = collections.OrderedDict((x[_SUBSCRIPTION_ID], x) for x in existing_ones)
+        else:
+            dic = collections.OrderedDict()
+
         dic.update((x[_SUBSCRIPTION_ID], x) for x in new_subscriptions)
         subscriptions = list(dic.values())
+        if subscriptions:
+            if active_one:
+                new_active_one = next(
+                    (x for x in new_subscriptions if x[_SUBSCRIPTION_ID] == active_subscription_id),
+                    None)
 
-        if active_one:
-            new_active_one = next(
-                (x for x in new_subscriptions if x[_SUBSCRIPTION_ID] == active_subscription_id),
-                None)
+                for s in subscriptions:
+                    s[_IS_DEFAULT_SUBSCRIPTION] = False
 
-            for s in subscriptions:
-                s[_IS_DEFAULT_SUBSCRIPTION] = False
-
-            if not new_active_one:
+                if not new_active_one:
+                    new_active_one = Profile._pick_working_subscription(new_subscriptions)
+            else:
                 new_active_one = Profile._pick_working_subscription(new_subscriptions)
-        else:
-            new_active_one = Profile._pick_working_subscription(new_subscriptions)
 
-        new_active_one[_IS_DEFAULT_SUBSCRIPTION] = True
-        default_sub_id = new_active_one[_SUBSCRIPTION_ID]
+            new_active_one[_IS_DEFAULT_SUBSCRIPTION] = True
+            default_sub_id = new_active_one[_SUBSCRIPTION_ID]
 
-        set_cloud_subscription(active_cloud.name, default_sub_id)
+            set_cloud_subscription(active_cloud.name, default_sub_id)
         self._storage[_SUBSCRIPTIONS] = subscriptions
 
     @staticmethod
@@ -422,6 +427,57 @@ class Profile(object):
                 str(account[_SUBSCRIPTION_ID]),
                 str(account[_TENANT_ID]))
 
+    def refresh_accounts(self, subscription_finder=None):
+        subscriptions = self.load_cached_subscriptions()
+        to_refresh = [s for s in subscriptions if s[_SUBSCRIPTION_NAME] != _MSI_ACCOUNT_NAME]
+        not_to_refresh = [s for s in subscriptions if s not in to_refresh]
+
+        from azure.cli.core._debug import allow_debug_adal_connection
+        allow_debug_adal_connection()
+        subscription_finder = subscription_finder or SubscriptionFinder(self.auth_ctx_factory,
+                                                                        self._creds_cache.adal_token_cache)
+        refreshed_list = set()
+        result = []
+        for s in to_refresh:
+            user_name = s[_USER_ENTITY][_USER_NAME]
+            if user_name in refreshed_list:
+                continue
+            refreshed_list.add(user_name)
+            is_service_principal = (s[_USER_ENTITY][_USER_TYPE] == _SERVICE_PRINCIPAL)
+            tenant = s[_TENANT_ID]
+            subscriptions = []
+            try:
+                if is_service_principal:
+                    sp_auth = ServicePrincipalAuth(self._creds_cache.retrieve_secret_of_service_principal(user_name))
+                    subscriptions = subscription_finder.find_from_service_principal_id(user_name, sp_auth, tenant,
+                                                                                       self._ad_resource_uri)
+                else:
+                    subscriptions = subscription_finder.find_from_user_account(user_name, None, None,
+                                                                               self._ad_resource_uri)
+            except Exception as ex:  # pylint: disable=broad-except
+                logger.warning("Refreshing for '%s' failed with an error '%s'. The existing accounts were not "
+                               "modified. You can run 'az login' later to explictly refresh them", user_name, ex)
+                result += deepcopy([r for r in to_refresh if r[_USER_ENTITY][_USER_NAME] == user_name])
+                continue
+
+            if not subscriptions:
+                if s[_SUBSCRIPTION_NAME] == _TENANT_LEVEL_ACCOUNT_NAME:
+                    subscriptions = Profile._build_tenant_level_accounts([s[_TENANT_ID]])
+
+                if not subscriptions:
+                    continue
+
+            consolidated = Profile._normalize_properties(subscription_finder.user_id,
+                                                         subscriptions,
+                                                         is_service_principal)
+            result += consolidated
+
+        if self._creds_cache.adal_token_cache.has_state_changed:
+            self._creds_cache.persist_cached_creds()
+
+        result = result + not_to_refresh
+        self._set_subscriptions(result, merge=False)
+
     def get_sp_auth_info(self, subscription_id=None, name=None, password=None, cert_file=None):
         from collections import OrderedDict
         account = self.get_subscription(subscription_id)
@@ -561,11 +617,11 @@ class SubscriptionFinder(object):
 
     def find_from_user_account(self, username, password, tenant, resource):
         context = self._create_auth_context(tenant)
-        token_entry = context.acquire_token_with_username_password(
-            resource,
-            username,
-            password,
-            _CLIENT_ID)
+        if password:
+            token_entry = context.acquire_token_with_username_password(resource, username, password, _CLIENT_ID)
+        else:  # when refresh account, we will leverage local cached tokens
+            token_entry = context.acquire_token(resource, username, _CLIENT_ID)
+
         self.user_id = token_entry[_TOKEN_ENTRY_USER_ID]
 
         if tenant is None:
diff --git a/src/command_modules/azure-cli-profile/HISTORY.rst b/src/command_modules/azure-cli-profile/HISTORY.rst
index 6d886b09f..a0d2c4d77 100644
--- a/src/command_modules/azure-cli-profile/HISTORY.rst
+++ b/src/command_modules/azure-cli-profile/HISTORY.rst
@@ -2,6 +2,10 @@
 
 Release History
 ===============
+(unreleased)
++++++++++++++++++++
+* account list: add `--refresh` to sycn up the latest subscriptions from server
+
 2.0.9 (2017-07-27)
 ++++++++++++++++++
 support login inside a VM with a managed identity
diff --git a/src/command_modules/azure-cli-profile/azure/cli/command_modules/profile/_params.py b/src/command_modules/azure-cli-profile/azure/cli/command_modules/profile/_params.py
index 4c2475a03..8a8b2eb98 100644
--- a/src/command_modules/azure-cli-profile/azure/cli/command_modules/profile/_params.py
+++ b/src/command_modules/azure-cli-profile/azure/cli/command_modules/profile/_params.py
@@ -27,4 +27,5 @@ register_cli_argument('logout', 'username', help='account user, if missing, logo
 
 register_cli_argument('account', 'subscription', options_list=('--subscription', '-s'), help='Name or ID of subscription.', completer=get_subscription_id_list)
 register_cli_argument('account list', 'all', help="List all subscriptions, rather just 'Enabled' ones", action='store_true')
+register_cli_argument('account list', 'refresh', help="retrieve up to date subscriptions from server", action='store_true')
 register_cli_argument('account show', 'show_auth_for_sdk', options_list=('--sdk-auth',), action='store_true', help='output result in compatible with Azure SDK auth file')
diff --git a/src/command_modules/azure-cli-profile/azure/cli/command_modules/profile/custom.py b/src/command_modules/azure-cli-profile/azure/cli/command_modules/profile/custom.py
index cbb7845dc..177be9eaf 100644
--- a/src/command_modules/azure-cli-profile/azure/cli/command_modules/profile/custom.py
+++ b/src/command_modules/azure-cli-profile/azure/cli/command_modules/profile/custom.py
@@ -14,15 +14,17 @@ from azure.cli.core.cloud import get_active_cloud
 logger = get_az_logger(__name__)
 
 
-def load_subscriptions(all_clouds=False):
+def load_subscriptions(all_clouds=False, refresh=False):
     profile = Profile()
+    if refresh:
+        subscriptions = profile.refresh_accounts()
     subscriptions = profile.load_cached_subscriptions(all_clouds)
     return subscriptions
 
 
-def list_subscriptions(all=False):  # pylint: disable=redefined-builtin
+def list_subscriptions(all=False, refresh=False):  # pylint: disable=redefined-builtin
     """List the imported subscriptions."""
-    subscriptions = load_subscriptions(all_clouds=all)
+    subscriptions = load_subscriptions(all_clouds=all, refresh=refresh)
     if not subscriptions:
         logger.warning('Please run "az login" to access your accounts.')
     for sub in subscriptions:


Test patch:
diff --git a/src/azure-cli-core/azure/cli/core/tests/test_profile.py b/src/azure-cli-core/azure/cli/core/tests/test_profile.py
index e65acb739..4a4d5fa91 100644
--- a/src/azure-cli-core/azure/cli/core/tests/test_profile.py
+++ b/src/azure-cli-core/azure/cli/core/tests/test_profile.py
@@ -9,6 +9,8 @@ import os
 import unittest
 import mock
 
+from copy import deepcopy
+
 from adal import AdalError
 from azure.mgmt.resource.subscriptions.models import (SubscriptionState, Subscription,
                                                       SubscriptionPolicies, SpendingLimit)
@@ -880,6 +882,81 @@ class Test_Profile(unittest.TestCase):
         mock_auth_context.acquire_token_with_client_certificate.assert_called_once_with(
             mgmt_resource, 'my app', mock.ANY, mock.ANY)
 
+    @mock.patch('adal.AuthenticationContext', autospec=True)
+    def test_refresh_accounts_one_user_account(self, mock_auth_context):
+        storage_mock = {'subscriptions': None}
+        profile = Profile(storage_mock, use_global_creds_cache=False)
+        consolidated = Profile._normalize_properties(self.user1, deepcopy([self.subscription1]), False)
+        profile._set_subscriptions(consolidated)
+        mock_auth_context.acquire_token_with_username_password.return_value = self.token_entry1
+        mock_auth_context.acquire_token.return_value = self.token_entry1
+        mock_arm_client = mock.MagicMock()
+        mock_arm_client.tenants.list.return_value = [TenantStub(self.tenant_id)]
+        mock_arm_client.subscriptions.list.return_value = deepcopy([self.subscription1, self.subscription2])
+        finder = SubscriptionFinder(lambda _, _2: mock_auth_context,
+                                    None,
+                                    lambda _: mock_arm_client)
+        # action
+        profile.refresh_accounts(finder)
+
+        # assert
+        result = storage_mock['subscriptions']
+        self.assertEqual(2, len(result))
+        self.assertEqual(self.id1.split('/')[-1], result[0]['id'])
+        self.assertEqual(self.id2.split('/')[-1], result[1]['id'])
+        self.assertTrue(result[0]['isDefault'])
+
+    @mock.patch('adal.AuthenticationContext', autospec=True)
+    def test_refresh_accounts_one_user_account_one_sp_account(self, mock_auth_context):
+        storage_mock = {'subscriptions': None}
+        profile = Profile(storage_mock, use_global_creds_cache=False)
+        sp_subscription1 = SubscriptionStub('sp-sub/3', 'foo-subname', self.state1, 'foo_tenant.onmicrosoft.com')
+        consolidated = Profile._normalize_properties(self.user1, deepcopy([self.subscription1]), False)
+        consolidated += Profile._normalize_properties('http://foo', [sp_subscription1], True)
+        profile._set_subscriptions(consolidated)
+        mock_auth_context.acquire_token_with_username_password.return_value = self.token_entry1
+        mock_auth_context.acquire_token.return_value = self.token_entry1
+        mock_auth_context.acquire_token_with_client_credentials.return_value = self.token_entry1
+        mock_arm_client = mock.MagicMock()
+        mock_arm_client.tenants.list.return_value = [TenantStub(self.tenant_id)]
+        mock_arm_client.subscriptions.list.side_effect = deepcopy([[self.subscription1], [self.subscription2, sp_subscription1]])
+        finder = SubscriptionFinder(lambda _, _2: mock_auth_context,
+                                    None,
+                                    lambda _: mock_arm_client)
+        profile._creds_cache.retrieve_secret_of_service_principal = lambda _: 'verySecret'
+        profile._creds_cache.flush_to_disk = lambda _: ''
+        # action
+        profile.refresh_accounts(finder)
+
+        # assert
+        result = storage_mock['subscriptions']
+        self.assertEqual(3, len(result))
+        self.assertEqual(self.id1.split('/')[-1], result[0]['id'])
+        self.assertEqual(self.id2.split('/')[-1], result[1]['id'])
+        self.assertEqual('3', result[2]['id'])
+        self.assertTrue(result[0]['isDefault'])
+
+    @mock.patch('adal.AuthenticationContext', autospec=True)
+    def test_refresh_accounts_with_nothing(self, mock_auth_context):
+        storage_mock = {'subscriptions': None}
+        profile = Profile(storage_mock, use_global_creds_cache=False)
+        consolidated = Profile._normalize_properties(self.user1, deepcopy([self.subscription1]), False)
+        profile._set_subscriptions(consolidated)
+        mock_auth_context.acquire_token_with_username_password.return_value = self.token_entry1
+        mock_auth_context.acquire_token.return_value = self.token_entry1
+        mock_arm_client = mock.MagicMock()
+        mock_arm_client.tenants.list.return_value = [TenantStub(self.tenant_id)]
+        mock_arm_client.subscriptions.list.return_value = []
+        finder = SubscriptionFinder(lambda _, _2: mock_auth_context,
+                                    None,
+                                    lambda _: mock_arm_client)
+        # action
+        profile.refresh_accounts(finder)
+
+        # assert
+        result = storage_mock['subscriptions']
+        self.assertEqual(0, len(result))
+
     @mock.patch('azure.cli.core._profile._load_tokens_from_file', autospec=True)
     def test_credscache_load_tokens_and_sp_creds_with_secret(self, mock_read_file):
         test_sp = {