Repository: getmoto/moto

Issue:
Can't find targets with instance tags in SSM send_command()
**moto**: 1.3.15.dev0
**boto**: 2.49.0
**boto3**: 1.12.35
**botocore**: 1.15.35

When using `SSM:send_command` with the `Targets` argument, no targets are found. Additionally when you try to run `SSM:get_command_invocation` on said command using an instance that should have been found (but wasn't in this case), it looks like boto3 is trying to contact AWS instead of being mocked by moto.

I'd expect the following test to pass:
```python
@mock_ec2
@mock_ssm
def test_get_command_invocations_by_instance_tag():
    ec2_client = boto3.client("ec2", region_name="us-east-1")
    tag_specifications = [{
        "ResourceType": "instance",
        "Tags": [{
            "Key": "Name",
            "Value": "test-tag"
        }]
    }]
    ec2_response = ec2_client.run_instances(ImageId="ami-1234abcd", MaxCount=1, MinCount=1,
                                            TagSpecifications=tag_specifications)
    instance_id = ec2_response["Instances"][0]["InstanceId"]

    ssm_client = boto3.client("ssm", region_name="us-east-1")
    ssm_response = ssm_client.send_command(DocumentName="AWS-RunShellScript",
                                           Targets=[{"Key": "tag:Name", "Values": ["test-tag"]}])
    target_count = ssm_response["Command"]["TargetCount"]
    assert target_count == 1

    command_id = ssm_response["Command"]["CommandId"]
    ssm_invocation = ssm_client.get_command_invocation(CommandId=command_id, InstanceId=instance_id)
    status = ssm_invocation["Status"]
    assert status == "Success"
```

Instead I get a huge amount of logging output that fills my terminal buffer, which makes me think maybe boto3 is not being mocked properly somewhere in this workflow e.g.:
```
botocore.loaders: DEBUG: Loading JSON file: /Users/****/.local/share/virtualenvs/moto-b58KUgnk/lib/python3.7/site-packages/botocore/data/endpoints.json
botocore.hooks: DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x10666ab90>
botocore.loaders: DEBUG: Loading JSON file: /Users/****/.local/share/virtualenvs/moto-b58KUgnk/lib/python3.7/site-packages/botocore/data/xray/2016-04-12/service-2.json
botocore.hooks: DEBUG: Event creating-client-class.xray: calling handler <function add_generate_presigned_url at 0x10663e170>
botocore.endpoint: DEBUG: Setting xray timeout as (60, 60)
botocore.loaders: DEBUG: Loading JSON file: /Users/****/.local/share/virtualenvs/moto-b58KUgnk/lib/python3.7/site-packages/botocore/data/_retry.json
botocore.client: DEBUG: Registering retry handlers for service: xray
botocore.hooks: DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
botocore.hooks: DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway
botocore.hooks: DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
botocore.hooks: DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
botocore.hooks: DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
botocore.hooks: DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
botocore.hooks: DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
botocore.hooks: DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
botocore.hooks: DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
botocore.hooks: DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
botocore.hooks: DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
botocore.credentials: DEBUG: Looking for credentials via: env
botocore.credentials: INFO: Found credentials in environment variables.
botocore.loaders: DEBUG: Loading JSON file: /Users/****/.local/share/virtualenvs/moto-b58KUgnk/lib/python3.7/site-packages/botocore/data/endpoints.json
botocore.hooks: DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x10666ab90>
botocore.loaders: DEBUG: Loading JSON file: /Users/****/.local/share/virtualenvs/moto-b58KUgnk/lib/python3.7/site-packages/botocore/data/ec2/2016-11-15/service-2.json
botocore.hooks: DEBUG: Event creating-client-class.ec2: calling handler <function add_generate_presigned_url at 0x10663e170>
botocore.endpoint: DEBUG: Setting ec2 timeout as (60, 60)
botocore.loaders: DEBUG: Loading JSON file: /Users/****/.local/share/virtualenvs/moto-b58KUgnk/lib/python3.7/site-packages/botocore/data/_retry.json
botocore.client: DEBUG: Registering retry handlers for service: ec2
botocore.hooks: DEBUG: Event before-parameter-build.ec2.RunInstances: calling handler <bound method ParameterAlias.alias_parameter_in_call of <botocore.handlers.ParameterAlias object at 0x10669ae90>>
```

This is followed by some POST calls:
```
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:ec2.us-east-1.amazonaws.com
x-amz-date:20200403T174038Z

content-type;host;x-amz-date
ad9ec84c115a4acbdef4ab6890b59d79241278f92e8e9b38d56de6c4832cb7b8
botocore.auth: DEBUG: StringToSign:
AWS4-HMAC-SHA256
20200403T174038Z
20200403/us-east-1/ec2/aws4_request
137bccef44ecbe8311c0aadf875ee580762af34cdfff92648cac534a8c8bf1e6
botocore.auth: DEBUG: Signature:
****
botocore.endpoint: DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ec2.us-east-1.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'Boto3/1.12.35 Python/3.7.7 Darwin/19.4.0 Botocore/1.15.35', 'X-Amz-Date': b'20200403T174038Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=foobar_key/20200403/us-east-1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=****', 'Content-Length': '196'}>
botocore.hooks: DEBUG: Event before-send.ec2.RunInstances: calling handler <moto.core.models.BotocoreStubber object at 0x104fde850>
botocore.parsers: DEBUG: Response headers: {'server': 'amazon.com'}
botocore.parsers: DEBUG: Response body:
b'<RunInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2013-10-15/"><requestId>59dbff89-35bd-4eac-99ed-be587EXAMPLE</requestId><reservationId>r-0c3b0997</reservationId><ownerId>123456789012</ownerId><groupSet><item><groupId>sg-245f6a01</groupId><groupName>default</groupName></item></groupSet><instancesSet><item><instanceId>i-171ad1d98d09d0dd1</instanceId><imageId>ami-1234abcd</imageId><instanceState><code>0</code><name>pending</name></instanceState><privateDnsName>ip-10-16-24-181.ec2.internal</privateDnsName><publicDnsName>ec2-54-214-166-21.compute-1.amazonaws.com</publicDnsName><dnsName>ec2-54-214-166-21.compute-1.amazonaws.com</dnsName><reason/><keyName>None</keyName><ebsOptimized>None</ebsOptimized><amiLaunchIndex>0</amiLaunchIndex><instanceType>m1.small</instanceType><launchTime>2020-04-03T17:40:38.000Z</launchTime><placement><availabilityZone>us-east-1a</availabilityZone><groupName/><tenancy>default</tenancy></placement><monitoring><state>enabled</state></monitoring><privateIpAddress>10.16.24.181</privateIpAddress><ipAddress>54.214.166.21</ipAddress><sourceDestCheck>true</sourceDestCheck><groupSet></groupSet><virtualizationType>paravirtual</virtualizationType><architecture>x86_64</architecture><kernelId>None</kernelId><clientToken/><hypervisor>xen</hypervisor><ebsOptimized>false</ebsOptimized><tagSet><item><key>Name</key><value>test-tag</value></item></tagSet><networkInterfaceSet><item><networkInterfaceId>eni-3708d6bb</networkInterfaceId><description>Primary network interface</description><ownerId>123456789012</ownerId><status>in-use</status><macAddress>1b:2b:3c:4d:5e:6f</macAddress><privateIpAddress>10.16.24.181</privateIpAddress><sourceDestCheck>true</sourceDestCheck><groupSet></groupSet><attachment><attachmentId>eni-attach-8f2a5ece</attachmentId><deviceIndex>0</deviceIndex><status>attached</status><attachTime>2015-01-01T00:00:00Z</attachTime><deleteOnTermination>true</deleteOnTermination></attachment><association><publicIp>54.214.166.21</publicIp><ipOwnerId>123456789012</ipOwnerId></association><privateIpAddressesSet><item><privateIpAddress>10.16.24.181</privateIpAddress><primary>true</primary><association><publicIp>54.214.166.21</publicIp><ipOwnerId>123456789012</ipOwnerId></association></item></privateIpAddressesSet></item></networkInterfaceSet></item></instancesSet></RunInstancesResponse>'
botocore.hooks: DEBUG: Event needs-retry.ec2.RunInstances: calling handler <botocore.retryhandler.RetryHandler object at 0x10faecdd0>
botocore.retryhandler: DEBUG: No retry needed.
botocore.hooks: DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x1048c9b90>
botocore.loaders: DEBUG: Loading JSON file: /Users/****/.local/share/virtualenvs/moto-b58KUgnk/lib/python3.7/site-packages/botocore/data/ssm/2014-11-06/service-2.json
botocore.hooks: DEBUG: Event creating-client-class.ssm: calling handler <function add_generate_presigned_url at 0x10489e170>
botocore.endpoint: DEBUG: Setting ssm timeout as (60, 60)
botocore.client: DEBUG: Registering retry handlers for service: ssm
botocore.hooks: DEBUG: Event before-parameter-build.ssm.SendCommand: calling handler <function generate_idempotent_uuid at 0x1048fbb00>
botocore.hooks: DEBUG: Event before-call.ssm.SendCommand: calling handler <function inject_api_version_header_if_needed at 0x1049035f0>
botocore.endpoint: DEBUG: Making request for OperationModel(name=SendCommand) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonSSM.SendCommand', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.12.35 Python/3.7.7 Darwin/19.4.0 Botocore/1.15.35'}, 'body': b'{"DocumentName": "AWS-RunShellScript", "Targets": [{"Key": "tag:Name", "Values": ["test-tag"]}]}', 'url': 'https://ssm.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x1109ecb10>, 'has_streaming_input': False, 'auth_type': None}}
botocore.hooks: DEBUG: Event request-created.ssm.SendCommand: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1109ec910>>
botocore.hooks: DEBUG: Event choose-signer.ssm.SendCommand: calling handler <function set_operation_specific_signer at 0x1048fb050>
botocore.auth: DEBUG: Calculating signature using v4 auth.
botocore.auth: DEBUG: CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ssm.us-east-1.amazonaws.com
x-amz-date:20200403T174038Z
x-amz-target:AmazonSSM.SendCommand

content-type;host;x-amz-date;x-amz-target
f62e612834faaa3dc015e7104f521eb85e5b683b95cb8086c0e4402f298d576d
botocore.auth: DEBUG: StringToSign:
AWS4-HMAC-SHA256
20200403T174038Z
20200403/us-east-1/ssm/aws4_request
7f314ca3afbc92aba2d375bbea5696f8de06d87174847fc40ee6819da898d911
botocore.auth: DEBUG: Signature:
****
botocore.endpoint: DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ssm.us-east-1.amazonaws.com/, headers={'X-Amz-Target': b'AmazonSSM.SendCommand', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.12.35 Python/3.7.7 Darwin/19.4.0 Botocore/1.15.35', 'X-Amz-Date': b'20200403T174038Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=foobar_key/20200403/us-east-1/ssm/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=****', 'Content-Length': '96'}>
botocore.hooks: DEBUG: Event before-send.ssm.SendCommand: calling handler <moto.core.models.BotocoreStubber object at 0x104fde850>
botocore.parsers: DEBUG: Response headers: {'server': 'amazon.com'}
botocore.parsers: DEBUG: Response body:
b'{"Command": {"CommandId": "123dde88-451a-4683-ab54-7056e4d2fa2b", "Comment": "", "CompletedCount": 0, "DocumentName": "AWS-RunShellScript", "ErrorCount": 0, "ExpiresAfter": "2020-04-03T11:40:38.256776", "InstanceIds": [], "MaxConcurrency": "50", "MaxErrors": "0", "NotificationConfig": {"NotificationArn": "string", "NotificationEvents": ["Success"], "NotificationType": "Command"}, "OutputS3Region": "", "OutputS3BucketName": "", "OutputS3KeyPrefix": "", "Parameters": {}, "RequestedDateTime": "2020-04-03T10:40:38.256776", "ServiceRole": "", "Status": "Success", "StatusDetails": "Details placeholder", "TargetCount": 0, "Targets": [{"Key": "tag:Name", "Values": ["test-tag"]}]}}'
botocore.hooks: DEBUG: Event needs-retry.ssm.SendCommand: calling handler <botocore.retryhandler.RetryHandler object at 0x1109ec2d0>
botocore.retryhandler: DEBUG: No retry needed.
--------------------- >> end captured logging << ---------------------
```

When I turn off log capture I see this failure:
```F
======================================================================
FAIL: test_ssm_boto3.test_get_command_invocations_by_instance_tag
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mcasiro/.local/share/virtualenvs/moto-b58KUgnk/lib/python3.7/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/Users/mcasiro/src/MISC/moto/moto/core/models.py", line 95, in wrapper
    result = func(*args, **kwargs)
  File "/Users/mcasiro/src/MISC/moto/moto/core/models.py", line 95, in wrapper
    result = func(*args, **kwargs)
  File "/Users/mcasiro/src/MISC/moto/tests/test_ssm/test_ssm_boto3.py", line 1606, in test_get_command_invocations_by_instance_tag
    assert target_count == 1
AssertionError:
    0 = {'Command': {'CommandId': '326bd0f7-012d-456a-95f6-17bd8adb618c', 'DocumentName': 'AWS-RunShellScript', 'Comment': '', 'ExpiresAfter': datetime.datetime(2020, 4, 3, 11, 29, 52, 676466), 'Parameters': {}, 'InstanceIds': [], 'Targets': [{'Key': 'tag:Name', 'Values': ['test-tag']}], 'RequestedDateTime': datetime.datetime(2020, 4, 3, 10, 29, 52, 676466), 'Status': 'Success', 'StatusDetails': 'Details placeholder', 'OutputS3Region': '', 'OutputS3BucketName': '', 'OutputS3KeyPrefix': '', 'MaxConcurrency': '50', 'MaxErrors': '0', 'TargetCount': 0, 'CompletedCount': 0, 'ErrorCount': 0, 'ServiceRole': '', 'NotificationConfig': {'NotificationArn': 'string', 'NotificationEvents': ['Success'], 'NotificationType': 'Command'}}, 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}["Command"]["TargetCount"]
>>  assert 0 == 1


----------------------------------------------------------------------
Ran 1 test in 0.448s

FAILED (failures=1)
```



Discussion:
Hi @Zullybur, the boto3 log output looks normal. Moto only mocks the actual HTTP requests, so any other processing done by boto3 will still happen, and will always have some log output.

You're right that we don't support the Targets-argument yet - will mark this as an enhancement to be picked up.

Thanks for providing a PR with the test! Always useful to have a starting point for people who want to work on this ticket.

Resolving patch:
diff --git a/moto/ssm/models.py b/moto/ssm/models.py
--- a/moto/ssm/models.py
+++ b/moto/ssm/models.py
@@ -6,12 +6,11 @@
 
 from moto.core import ACCOUNT_ID, BaseBackend, BaseModel
 from moto.core.exceptions import RESTError
-from moto.cloudformation import cloudformation_backends
+from moto.ec2 import ec2_backends
 
 import datetime
 import time
 import uuid
-import itertools
 import json
 import yaml
 import hashlib
@@ -246,9 +245,6 @@ def __init__(
         if targets is None:
             targets = []
 
-        self.error_count = 0
-        self.completed_count = len(instance_ids)
-        self.target_count = len(instance_ids)
         self.command_id = str(uuid.uuid4())
         self.status = "Success"
         self.status_details = "Details placeholder"
@@ -262,7 +258,6 @@ def __init__(
 
         self.comment = comment
         self.document_name = document_name
-        self.instance_ids = instance_ids
         self.max_concurrency = max_concurrency
         self.max_errors = max_errors
         self.notification_config = notification_config
@@ -274,14 +269,19 @@ def __init__(
         self.targets = targets
         self.backend_region = backend_region
 
-        # Get instance ids from a cloud formation stack target.
-        stack_instance_ids = [
-            self.get_instance_ids_by_stack_ids(target["Values"])
-            for target in self.targets
-            if target["Key"] == "tag:aws:cloudformation:stack-name"
-        ]
-
-        self.instance_ids += list(itertools.chain.from_iterable(stack_instance_ids))
+        self.instance_ids = instance_ids
+        self.instance_ids += self._get_instance_ids_from_targets()
+        # Ensure no duplicate instance_ids
+        self.instance_ids = list(set(self.instance_ids))
+
+        # NOTE: All of these counts are 0 in the ssm:SendCommand response
+        # received from a real AWS backend.  The counts are correct when
+        # making subsequent calls to ssm:DescribeCommand or ssm:ListCommands.
+        # Not likely to cause any problems, but perhaps an area for future
+        # improvement.
+        self.error_count = 0
+        self.completed_count = len(instance_ids)
+        self.target_count = len(instance_ids)
 
         # Create invocations with a single run command plugin.
         self.invocations = []
@@ -290,19 +290,15 @@ def __init__(
                 self.invocation_response(instance_id, "aws:runShellScript")
             )
 
-    def get_instance_ids_by_stack_ids(self, stack_ids):
-        instance_ids = []
-        cloudformation_backend = cloudformation_backends[self.backend_region]
-        for stack_id in stack_ids:
-            stack_resources = cloudformation_backend.list_stack_resources(stack_id)
-            instance_resources = [
-                instance.id
-                for instance in stack_resources
-                if instance.type == "AWS::EC2::Instance"
-            ]
-            instance_ids.extend(instance_resources)
-
-        return instance_ids
+    def _get_instance_ids_from_targets(self):
+        target_instance_ids = []
+        ec2_backend = ec2_backends[self.backend_region]
+        ec2_filters = {target["Key"]: target["Values"] for target in self.targets}
+        reservations = ec2_backend.all_reservations(filters=ec2_filters)
+        for reservation in reservations:
+            for instance in reservation.instances:
+                target_instance_ids.append(instance.id)
+        return target_instance_ids
 
     def response_object(self):
         r = {


Test patch:
diff --git a/tests/test_ssm/test_ssm_boto3.py b/tests/test_ssm/test_ssm_boto3.py
--- a/tests/test_ssm/test_ssm_boto3.py
+++ b/tests/test_ssm/test_ssm_boto3.py
@@ -11,7 +11,7 @@
 from botocore.exceptions import ClientError, ParamValidationError
 from nose.tools import assert_raises
 
-from moto import mock_ssm
+from moto import mock_ec2, mock_ssm
 
 
 @mock_ssm
@@ -1713,3 +1713,36 @@ def test_get_command_invocation():
         invocation_response = client.get_command_invocation(
             CommandId=cmd_id, InstanceId=instance_id, PluginName="FAKE"
         )
+
+
+@mock_ec2
+@mock_ssm
+def test_get_command_invocations_by_instance_tag():
+    ec2 = boto3.client("ec2", region_name="us-east-1")
+    ssm = boto3.client("ssm", region_name="us-east-1")
+    tag_specifications = [
+        {"ResourceType": "instance", "Tags": [{"Key": "Name", "Value": "test-tag"}]}
+    ]
+    num_instances = 3
+    resp = ec2.run_instances(
+        ImageId="ami-1234abcd",
+        MaxCount=num_instances,
+        MinCount=num_instances,
+        TagSpecifications=tag_specifications,
+    )
+    instance_ids = []
+    for instance in resp["Instances"]:
+        instance_ids.append(instance["InstanceId"])
+    instance_ids.should.have.length_of(num_instances)
+
+    command_id = ssm.send_command(
+        DocumentName="AWS-RunShellScript",
+        Targets=[{"Key": "tag:Name", "Values": ["test-tag"]}],
+    )["Command"]["CommandId"]
+
+    resp = ssm.list_commands(CommandId=command_id)
+    resp["Commands"][0]["TargetCount"].should.equal(num_instances)
+
+    for instance_id in instance_ids:
+        resp = ssm.get_command_invocation(CommandId=command_id, InstanceId=instance_id)
+        resp["Status"].should.equal("Success")