Repository: localstack/localstack

Issue:
Service "s3" not yet available, retrying...
Hello there

After installing localstack and trying to start several services on my machine, s3 always failed to start
The command I am using to start them up is

`SERVICES=sqs,sns,s3,lambda DEBUG=1 localstack start`
(With `DEBUG=1` in place already for debugging)

First few lines of the output are:

```
2018-06-19T10:05:57:WARNING:infra.py: Service "s3" not yet available, retrying...
2018-06-19T10:06:00:WARNING:infra.py: Service "s3" not yet available, retrying...
2018-06-19T10:06:05:WARNING:infra.py: Service "s3" not yet available, retrying...
2018-06-19T10:06:08:WARNING:infra.py: Service "s3" not yet available, retrying...
2018-06-19T10:06:12:WARNING:infra.py: Service "s3" not yet available, retrying...
2018-06-19T10:06:15:WARNING:infra.py: Service "s3" not yet available, retrying...
2018-06-19T10:06:19:WARNING:infra.py: Service "s3" not yet available, retrying...
2018-06-19T10:06:22:ERROR:localstack.services.s3.s3_starter: S3 health check failed: An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/localstack/services/infra.py", line 344, in check_infra
    raise e
  File "/usr/local/lib/python3.6/site-packages/localstack/services/infra.py", line 341, in check_infra
    plugin.check(expect_shutdown=expect_shutdown, print_error=print_error)
  File "/usr/local/lib/python3.6/site-packages/localstack/services/infra.py", line 80, in check
    return self.check_function(expect_shutdown=expect_shutdown, print_error=print_error)
  File "/usr/local/lib/python3.6/site-packages/localstack/services/s3/s3_starter.py", line 23, in check_s3
    assert isinstance(out['Buckets'], list)
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:
...
```

I have been trying to tackle this problem for a few hours already, without any success, I tried the latest verion of localstack as well as 0.8.6.1 which works on another machine

I am installing it trough pip (`pip install localstack`)

Thanks for any help in advance!



┆Issue is synchronized with this [Jira Bug](https://localstack.atlassian.net/browse/LOC-309) by [Unito](https://www.unito.io/learn-more)



Discussion:
Facing same issue.
I experienced this issue as well.  I was able to resolve it by reinstalling Python.
Same issue, but I've got too much Python installed to "reinstall" Python to fix it.
same issue here, anyone knows how to fix it?
I am using docker and the issue exists. Sadly every service is in a single container and when i start it all with docker compose the S3 service does not start. 

However when i start only the S3 service in the container it works...

**environment:**
      - SERVICES=s3
I was able to root cause this issue with newer versions of localstack. 

Environment details:
Python version : 3.7
localstack version : 0.9.1
localstack is installed via pip3
localstack running inside a docker container, which contained both python2 as well as python3 installations.

With respect to S3, the source code is bringing up an instance of python within the container

```
def start_s3(port=None, backend_port=None, asynchronous=None, update_listener=None):
    port = port or config.PORT_S3
    backend_port = DEFAULT_PORT_S3_BACKEND
    cmd = 'python "%s" s3 -p %s -H 0.0.0.0' % (__file__, backend_port)
    print('Starting mock S3 (%s port %s)...' % (get_service_protocol(), port))
    start_proxy_for_service('s3', port, backend_port, update_listener)
    env_vars = {'PYTHONPATH': ':'.join(sys.path)}
    return do_run(cmd, asynchronous, env_vars=env_vars)
```

We can see that it just runs `python ......`. In my docker container, I had both python3 as well as python2 installed. localstack was installed by using pip3. So all dependencies for localstack were installed for python3. 

However, the `python` command was pointing towards my python2 installation, and hence, the python process started by localstack failed to come up because it was unable to find any of the dependencies required and it failed silently.

To circumvent the issue, I just installed localstack within a venv and brought up S3 within that venv (after sourcing the venv) to make sure that running `python` points towards the correct version of `python` for which localstack dependencies were installed etc.

TL;DR make sure that localstack and its dependencies are installed for the right python version and make sure that running `python` points towards the right python version as well.
I had the same issue for a while on macos, but downgraded localstack to 0.9.0 and downgraded python to 3.6.0 and it started working.
Setting up everything fresh, and getting this error as well.  S3 is the only service that is failing:
```
∴ python3 --version
Python 3.7.3
∴ python --version
Python 2.7.10
∴ localstack -v
0.9.4
```
As per @prachetaa's comment I then set `python` to point to `python3` (& same for `pip` => `pip3`):
```
unlink /usr/local/bin/python
ln -s /usr/local/bin/python3 /usr/local/bin/python
unlink /usr/local/bin/pip
ln -s /usr/local/bin/pip3 /usr/local/bin/pip
```
And now I no longer have the problem, and I didn't have to downgrade anything, cc @eichelkrauta

Although now it is failing the same way for `dynamodb`... fml

Using `DEBUG=1`:
```
2019-06-01T07:29:19:WARNING:infra.py: Service "dynamodb" not yet available, retrying...
2019-06-01T07:29:22:INFO:werkzeug: 127.0.0.1 - - [01/Jun/2019 07:29:22] "GET / HTTP/1.1" 200 -
[2019-06-01T07:29:31,314][WARN ][o.e.c.r.a.DiskThresholdMonitor] [_zGq4tp] flood stage disk watermark [95%] exceeded on [_zGq4tp3TuSIAteokT2fZg][_zGq4tp][/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/localstack/infra/elasticsearch/data/nodes/0] free: 29.9gb[3.2%], all indices on this node will marked read-only
2019-06-01T07:29:33:ERROR:localstack.services.dynamodb.dynamodb_starter: DynamoDB health check failed:  Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/localstack/services/infra.py", line 411, in check_infra
    raise e
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/localstack/services/infra.py", line 408, in check_infra
    plugin.check(expect_shutdown=expect_shutdown, print_error=print_error)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/localstack/services/infra.py", line 93, in check
    return self.check_function(expect_shutdown=expect_shutdown, print_error=print_error)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/localstack/services/dynamodb/dynamodb_starter.py", line 30, in check_dynamodb
    assert isinstance(out['TableNames'], list)
TypeError: 'NoneType' object is not subscriptable
```
I think this other part of the debug log may reference the real issue:
```
[2019-06-01T07:38:16,174][WARN ][o.e.c.r.a.DiskThresholdMonitor] [FaRbcgC] flood stage disk watermark [95%] exceeded on [FaRbcgCdQ16tDD80Aalx_A][FaRbcgC][/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/localstack/infra/elasticsearch/data/nodes/0] free: 29.2gb[3.1%], all indices on this node will marked read-only
```
Thankfully, as I don't use dynamodb on this project:
```
SERVICES=sqs,s3 DEBUG=1 localstack start
```
This at least is no longer crashing out of localstack.
I checked out the code and ran `make clean install test`, and got some errors, one of which is:
```
======================================================================
FAIL: test_lambda_streams_batch_and_transactions (tests.integration.test_integration.IntegrationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/peterboling/Documents/src/aws/localstack/tests/integration/test_integration.py", line 428, in test_lambda_streams_batch_and_transactions
    retry(check_events, retries=9, sleep=3)
  File "/Users/peterboling/Documents/src/aws/localstack/localstack/utils/common.py", line 395, in retry
    raise raise_error
  File "/Users/peterboling/Documents/src/aws/localstack/localstack/utils/common.py", line 391, in retry
    return function(**kwargs)
  File "/Users/peterboling/Documents/src/aws/localstack/tests/integration/test_integration.py", line 398, in check_events
    self.assertEqual(len(events), num_events)
AssertionError: 0 != 20
-------------------- >> begin captured logging << --------------------
KCL: CRITICAL: Jun 01, 2019 10:13:27 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownTask call
KCL: CRITICAL: SEVERE: Caught exception:
KCL: WARNING: WARNING: Shutdown was called but this processor is already shutdown. Not doing anything.
KCL: WARNING: Jun 01, 2019 10:13:30 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker info
KCL: WARNING: Jun 01, 2019 10:13:31 AM com.amazonaws.http.AmazonHttpClient createSocketFactoryRegistry
KCL: WARNING: WARNING: SSL Certificate checking for endpoints has been explicitly disabled.
KCL: WARNING: Jun 01, 2019 10:13:32 AM com.amazonaws.http.AmazonHttpClient createSocketFactoryRegistry
KCL: WARNING: WARNING: SSL Certificate checking for endpoints has been explicitly disabled.
KCL: WARNING: Jun 01, 2019 10:13:32 AM com.amazonaws.http.AmazonHttpClient createSocketFactoryRegistry
KCL: WARNING: WARNING: SSL Certificate checking for endpoints has been explicitly disabled.
KCL: WARNING: Jun 01, 2019 10:13:32 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker build
KCL: WARNING: WARNING: Received configuration for both region name as us-east-1, and Amazon Kinesis endpoint as http://localhost:4568. Amazon Kinesis endpoint will overwrite region name.
KCL: WARNING: WARNING: An illegal reflective access operation has occurred
KCL: WARNING: WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:/Users/peterboling/Documents/src/aws/localstack/.venv/lib/python3.7/site-packages/amazon_kclpy/jars/jackson-databind-2.6.7.1.jar) to field java.lang.Throwable.cause
KCL: WARNING: WARNING: Please consider reporting this to the maintainers of com.fasterxml.jackson.databind.util.ClassUtil
KCL: WARNING: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
KCL: WARNING: WARNING: All illegal access operations will be denied in a future release
KCL: WARNING: Jun 01, 2019 10:13:32 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibLeaseCoordinator initialize
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
l_3454c077: INFO: Lambda log message - logging module
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
tests.integration.test_integration: WARNING: DynamoDB updates retrieved (actual/expected): 0/20
KCL: CRITICAL: Jun 01, 2019 10:13:55 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker runProcessLoop
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
tests.integration.test_integration: WARNING: DynamoDB updates retrieved (actual/expected): 0/20
KCL: CRITICAL: Jun 01, 2019 10:13:59 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker runProcessLoop
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
tests.integration.test_integration: WARNING: DynamoDB updates retrieved (actual/expected): 0/20
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
KCL: CRITICAL: Jun 01, 2019 10:14:03 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker runProcessLoop
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
tests.integration.test_integration: WARNING: DynamoDB updates retrieved (actual/expected): 0/20
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
tests.integration.test_integration: WARNING: DynamoDB updates retrieved (actual/expected): 0/20
KCL: CRITICAL: Jun 01, 2019 10:14:07 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker runProcessLoop
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
tests.integration.test_integration: WARNING: DynamoDB updates retrieved (actual/expected): 0/20
KCL: CRITICAL: Jun 01, 2019 10:14:11 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker runProcessLoop
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
tests.integration.test_integration: WARNING: DynamoDB updates retrieved (actual/expected): 0/20
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
KCL: CRITICAL: Jun 01, 2019 10:14:15 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker runProcessLoop
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
tests.integration.test_integration: WARNING: DynamoDB updates retrieved (actual/expected): 0/20
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
tests.integration.test_integration: WARNING: DynamoDB updates retrieved (actual/expected): 0/20
KCL: CRITICAL: Jun 01, 2019 10:14:19 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker runProcessLoop
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
KCL: CRITICAL: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
tests.integration.test_integration: WARNING: DynamoDB updates retrieved (actual/expected): 0/20
KCL: CRITICAL: Jun 01, 2019 10:14:23 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker runProcessLoop
KCL: CRITICAL: SEVERE: Worker.run caught exception, sleeping for 1000 milli seconds!
--------------------- >> end captured logging << ---------------------
```

In spite of the test suite errors, when I run `make infra` the parts that I need to work (s3 & sqs) seem to work!

Resolving patch:
diff --git a/localstack/services/s3/s3_starter.py b/localstack/services/s3/s3_starter.py
--- a/localstack/services/s3/s3_starter.py
+++ b/localstack/services/s3/s3_starter.py
@@ -35,7 +35,7 @@ def check_s3(expect_shutdown=False, print_error=False):
 def start_s3(port=None, backend_port=None, asynchronous=None, update_listener=None):
     port = port or config.PORT_S3
     backend_port = DEFAULT_PORT_S3_BACKEND
-    cmd = 'python "%s" s3 -p %s -H 0.0.0.0' % (__file__, backend_port)
+    cmd = '%s "%s" s3 -p %s -H 0.0.0.0' % (sys.executable, __file__, backend_port)
     print('Starting mock S3 (%s port %s)...' % (get_service_protocol(), port))
     start_proxy_for_service('s3', port, backend_port, update_listener)
     env_vars = {'PYTHONPATH': ':'.join(sys.path)}


Test patch: