Repository: hylang/hy

Issue:
rply causes a crash on read-only filesystems
I currently have the following file setup on AWS Lambda:

```
challenge
├── handlers.py
├── hello.hy
├── __init__.hy
└── __init__.py
```

handlers.py
```python
import hy
from hello import hello
```

hello.hy
```clojure
(import [json [dumps]])

(defn hello [event context]
  (setv body {"message" "Go Serverless v1.0! Your function executed successfully!"
              "input" event})
  ({"statusCode" 200
    "body" (dumps body)}))
```

AWS lambda calls the `hello` function inside `handlers.py`. My objective was to use this file as a proxy for writing aws lambda functions with hylang. But I keep getting this error when executing it in production:

```
module initialization error
[Errno 30] Read-only file system: '/home/sbx_user1059'
```

(the number 1059 varies)

I think this is due to the AST compiling on importing `hy` in `handlers.py`. AWS lambda does not permit that directory to be written.

Any way that this could be circumvented?


Discussion:
You can disable bytecode compilation by setting the environment variable PYTHONDONTWRITEBYTECODE. The real bug here is that we're ignoring `sys.dont_write_bytecode`, which is set via `python -B`, which AWS probably uses. 
Still got the same error :/
I don't think byte-compilation should be the issue, because Hy's call to `write_code_as_pyc` is wrapped in a `try` that catches `IOError` and `OSError`. The quoted error message doesn't include a stack trace, but the presence of an errno suggests it's an `OSError`.

By the way, @daniel-severo, this probably isn't your problem here, but it's a Bad Idea to have a Python module and a Hy module with the same name in the same directory, especially when that name is `__init__`.
> The quoted error message doesn't include a stack trace

The first thing to do here is probably to see how to convince AWS Lambda to give you the stack trace.
@Kodiologist yeah, I'm gonna remove one of them. I'll try to investigate it further.
@kirbyfan64 not respecting `sys.dont_write_bytecode` is fixed in #1518 
Looks like the bytecode generation wasn't the problem. For some reason, after generating python code with `hy2py`, this import statement is the problem:

```python
from hy.core.language import complement, distinct, drop_while, first, inc, iterate, last, reduce, take_while
```

Maybe something in the [importer](https://github.com/hylang/hy/blob/master/hy/importer.py) is trying to write data somewhere.
Did you get a traceback? https://docs.aws.amazon.com/lambda/latest/dg/python-exceptions.html
I was unable to get any more logging information from aws lambda. 

Although, strangely enough, ti looks like it breaks BEFORE the actual import statement.

```python
print("before everything")
from hy.core.language import complement, distinct, drop_while, first, inc, iterate, last, reduce, take_while
```

```
module initialization error
[Errno 30] Read-only file system: '/home/sbx_user1073'
before everything
```

I was able to get a stack trace by putting the `import hy` inside the function called by AWS Lambda instead of the mainline code.<details>
```{
  "errorMessage": "[Errno 30] Read-only file system: '/home/sbx_user1065'",
  "errorType": "OSError",
  "stackTrace": [
    [
      "/var/task/handlers.py",
      2,
      "hello",
      "import hy"
    ],
    [
      "/var/task/hy/__init__.py",
      11,
      "<module>",
      "import hy.importer  # NOQA"
    ],
    [
      "/var/task/hy/importer.py",
      7,
      "<module>",
      "from hy.compiler import hy_compile, HyTypeError"
    ],
    [
      "/var/task/hy/compiler.py",
      11,
      "<module>",
      "from hy.lex.parser import mangle"
    ],
    [
      "/var/task/hy/lex/__init__.py",
      9,
      "<module>",
      "from hy.lex.parser import parser"
    ],
    [
      "/var/task/hy/lex/parser.py",
      374,
      "<module>",
      "parser = pg.build()"
    ],
    [
      "/var/task/rply/parsergenerator.py",
      194,
      "build",
      "os.makedirs(cache_dir, mode=0o0700)"
    ],
    [
      "/var/lang/lib/python3.6/os.py",
      210,
      "makedirs",
      "makedirs(head, mode, exist_ok)"
    ],
    [
      "/var/lang/lib/python3.6/os.py",
      210,
      "makedirs",
      "makedirs(head, mode, exist_ok)"
    ],
    [
      "/var/lang/lib/python3.6/os.py",
      220,
      "makedirs",
      "mkdir(name, mode)"
    ]
  ]
}
```
</details>

It seems that the guilty code is in `rply.parsergenerator`, which tries to write out a cache of the generated parser without catching any errors. I've opened an issue at alex/rply#74.
We can write to /tmp on AWS lambda ... We could have a way to configure where is the cache located (independently of the fact that it should write on read-only fs ..)
@daniel-severo Did upgrading `rply` solve your problem?
Sort of. It enabled me to deploy my code to aws lambda as long as I don't use anything from the shadow library. For some reason it acusses that the module isn't available.

Since I use toolz most of the time, this wasn't such a big issue.
That sounds like it could be another bug. Try opening a new issue for it. Let's keep this one open till a new rply is out.

Resolving patch:
diff --git a/hy/lex/parser.py b/hy/lex/parser.py
--- a/hy/lex/parser.py
+++ b/hy/lex/parser.py
@@ -17,10 +17,7 @@
 from .exceptions import LexException, PrematureEndOfInput
 
 
-pg = ParserGenerator(
-    [rule.name for rule in lexer.rules] + ['$end'],
-    cache_id="hy_parser"
-)
+pg = ParserGenerator([rule.name for rule in lexer.rules] + ['$end'])
 
 mangle_delim = 'X'
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -30,7 +30,7 @@ def run(self):
                         "." + filename[:-len(".hy")])
         install.run(self)
 
-install_requires = ['rply>=0.7.5', 'astor', 'funcparserlib>=0.3.6', 'clint>=0.4']
+install_requires = ['rply>=0.7.6', 'astor', 'funcparserlib>=0.3.6', 'clint>=0.4']
 if os.name == 'nt':
     install_requires.append('pyreadline>=2.1')
 


Test patch: