Serverless: Python - virtualenv - { "errorMessage": "Unable to import module 'handler'" }
Date: 2017-08-06 19:03:30

I've been using the https://serverless.com/[Serverless] library to deploy and run some Python functions on AWS lambda recently and was initially confused about how to handle my dependencies.

I tend to create a new https://virtualenv.pypa.io/en/stable/[virtualenv] for each of my project so let's get that setup first:

== Prerequisites

[source,bash]
----

$ npm install serverless
----

[source,bash]
----

$ virtualenv -p python3 a
$ . a/bin/activate
----

Now let's create our Serverless project. I'm going to install the http://docs.python-requests.org/en/master/[requests] library so that I can use it in my function.

== My Serverless project

_serverless.yaml_

[source,yaml]
----

service: python-starter-template

frameworkVersion: ">=1.2.0 <2.0.0"

provider:
  name: aws
  runtime: python3.6
  timeout: 180

functions:
  starter-function:
      name: Starter
      handler: handler.starter
----

_handler.py_

[source,python]
----

import requests

def starter(event, context):
    print("event:", event, "context:", context)
    r = requests.get("http://www.google.com")
    print(r.status_code)
----

[source,bash]
----

$ pip install requests
----

Ok, we're now ready to try out the function. A nice feature of Serverless is that it lets us try out functions locally before we deploy them onto one of the Cloud providers:

[source,bash]
----

$ ./node_modules/serverless/bin/serverless invoke local --function starter-function
event: {} context: <__main__.FakeLambdaContext object at 0x10bea9a20>
200
null
----

So far so good. Next we'll deploy our function to AWS. I'm assuming you've already got your credentials setup but if not you can follow the https://serverless.com/framework/docs/providers/aws/guide/credentials/[tutorial on the Serverless page].

[source,bash]
----

$ ./node_modules/serverless/bin/serverless deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (26.48 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.........
Serverless: Stack update finished...
Service Information
service: python-starter-template
stage: dev
region: us-east-1
api keys:
  None
endpoints:
  None
functions:
  starter-function: python-starter-template-dev-starter-function
----

Now let's invoke our function:

[source,bash]
----

$ ./node_modules/serverless/bin/serverless invoke --function starter-function
{
    "errorMessage": "Unable to import module 'handler'"
}

  Error --------------------------------------------------

  Invoked function failed

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           6.7.0
     Serverless Version:     1.19.0
----

Hmmm, that's odd - I wonder why it can't import our handler module? We can call the logs function to check. The logs are usually a few seconds behind so we'll have to be a bit patient if we don't see them immediately.

[source,bash]
----

$ ./node_modules/serverless/bin/serverless logs  --function starter-function
START RequestId: 735efa84-7ad0-11e7-a4ef-d5baf0b46552 Version: $LATEST
Unable to import module 'handler': No module named 'requests'

END RequestId: 735efa84-7ad0-11e7-a4ef-d5baf0b46552
REPORT RequestId: 735efa84-7ad0-11e7-a4ef-d5baf0b46552	Duration: 0.42 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 22 MB
----

That explains it - the requests module wasn't imported.

If we look in +++<cite>+++.serverless/python-starter-template.zip</p> we can see that the requests module is hidden inside the +++<cite>+++a+++</cite>+++ directory and the instance of Python that runs on Lambda doesn't know where to find it. </p>

I'm sure there are other ways of solving this but the easiest one I found is a Serverless plugin called https://www.npmjs.com/package/serverless-python-requirements[serverless-python-requirements].

So how does this plugin work?

____
A Serverless v1.x plugin to automatically bundle dependencies from requirements.txt and make them available in your PYTHONPATH.
____

Doesn't sound too tricky - we can use +++<cite>+++pip freeze+++</cite>+++ to get our list of requirements and write them into a file. Let's rework +++<cite>+++serverless.yaml+++</cite>+++ to make use of the plugin:

== My Serverless project using serverless-python-requirements ~~~bash $ npm install --save serverless-python-requirements ~~~ ~~~bash $ pip freeze > requirements.txt $ cat requirements.txt certifi==2017.7.27.1 chardet==3.0.4 idna==2.5 requests==2.18.3 urllib3==1.22 ~~~

_serverless.yaml_ ~~~yaml service: python-starter-template frameworkVersion: ">=1.2.0 <2.0.0" provider: name: aws runtime: python3.6 timeout: 180 plugins: - serverless-python-requirements functions: starter-function: name: Starter handler: handler.starter package: exclude: - a/** # virtualenv ~~~

We have two changes from before:

* We added the +++<cite>+++serverless-python-requirements+++</cite>+++ plugin
* We excluded the +++<cite>+++a+++</cite>+++ directory since we don't need it

Let's deploy again and run the function: ~~~bash $ ./node_modules/serverless/bin/serverless deploy Serverless: Parsing Python requirements.txt Serverless: Installing required Python packages for runtime python3.6\... Serverless: Linking required Python packages\... Serverless: Packaging service\... Serverless: Excluding development dependencies\... Serverless: Unlinking required Python packages\... Serverless: Uploading CloudFormation file to S3\... Serverless: Uploading artifacts\... Serverless: Uploading service .zip file to S3 (14.39 MB)\... Serverless: Validating template\... Serverless: Updating Stack\... Serverless: Checking Stack update progress\... \...\...\... Serverless: Stack update finished\... Service Information service: python-starter-template stage: dev region: us-east-1 api keys: None endpoints: None functions: starter-function: python-starter-template-dev-starter-function ~~~ ~~~bash $ ./node_modules/serverless/bin/serverless invoke --function starter-function null ~~~

Looks good. Let's check the logs: ~~~bash $ ./node_modules/serverless/bin/serverless logs --function starter-function START RequestId: 61e8eda7-7ad4-11e7-8914-03b8a7793a24 Version: $LATEST event: {} context: <__main__.LambdaContext object at 0x7f568b105f28> 200 END RequestId: 61e8eda7-7ad4-11e7-8914-03b8a7793a24 REPORT RequestId: 61e8eda7-7ad4-11e7-8914-03b8a7793a24 Duration: 55.55 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 29 M ~~~

All good here as well so we're done! +++</cite>+++