travis: use docker build env

Also limit test memory usage. The docker build env likely reports the full memory amount of the physical host while the container itself is actually limited to 4gb, causing the jvm to gobble up ram and eventually getting killed.