Repository: exasol/integration-test-docker-environment (language: python)

Issue:
Index out of range exception in FileDirectoryListHasher 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/base/base_task.py", line 224, in run
>     task_generator = self.run_task()
>   File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/docker_image_analyze_task.py", line 143, in run_task
>     .generate_image_hash(image_info_of_dependencies)
>   File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/utils/build_context_hasher.py", line 18, in generate_image_hash
>     hash_of_build_context = self._generate_build_context_hash()
>   File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/utils/build_context_hasher.py", line 33, in _generate_build_context_hash
>     hash_of_build_context = files_directories_list_hasher.hash(files_directories_to_hash)
>   File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py", line 72, in hash
>     file_or_directory, collected_directories, collected_files)
>   File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py", line 135, in collect_files_and_directories
>     numCharacters += sum([len(d[1]) for d in new_directories])
>   File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py", line 135, in <listcomp>
>     numCharacters += sum([len(d[1]) for d in new_directories])
> IndexError: string index out of range

https://github.com/exasol/integration-test-docker-environment/blob/84c4e2d3b9a8cfd82dce655577d340a399b63b8b/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py#L135


Discussion / hints:

Fix patch:
diff --git a/doc/changes/changelog.md b/doc/changes/changelog.md
index d658aa68a..a542f888c 100644
--- a/doc/changes/changelog.md
+++ b/doc/changes/changelog.md
@@ -1,5 +1,6 @@
 # Changes
 
+* [0.4.1](changes_0.4.1.md)
 * [0.4.0](changes_0.4.0.md)
 * [0.3.1](changes_0.3.1.md)
 * [0.3.0](changes_0.3.0.md)
diff --git a/doc/changes/changes_0.4.1.md b/doc/changes/changes_0.4.1.md
new file mode 100644
index 000000000..d9400774c
--- /dev/null
+++ b/doc/changes/changes_0.4.1.md
@@ -0,0 +1,29 @@
+# Integration-Test-Docker-Environment 0.4.1, released 2021-07-29
+
+Code name: Fix index out-of-range in FileDirectoryListHasher
+
+## Summary
+
+This release fixes an index-out-of-range bug in the FileDirectoryListHasher which occurs for single character file or directory names.
+
+### Supported Exasol Versions
+
+* **6.2**: up to 6.2.15
+* **7.0**: up to 7.0.11
+
+If you need further versions, please open an issue.
+
+### Tested Docker Runtimes
+
+- Docker Default Runtime
+- [NVIDIA Container Runtime](https://github.com/NVIDIA/nvidia-container-runtime) for GPU accelerated UDFs
+
+## Bug Fixes:
+   - #103: Index out of range exception in FileDirectoryListHasher
+
+## Features / Enhancements:
+n/a  
+
+## Refactoring:
+n/a  
+
diff --git a/pyproject.toml b/pyproject.toml
index f847ef5de..e8de308c5 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "exasol-integration-test-docker-environment"
-version = "0.4.0"
+version = "0.4.1"
 description = "Integration Test Docker Environment for Exasol"
 
 license = "MIT"
diff --git a/setup.py b/setup.py
index 4a45ce0ce..445b9838f 100644
--- a/setup.py
+++ b/setup.py
@@ -72,7 +72,7 @@
 
 setup_kwargs = {
     'name': 'exasol-integration-test-docker-environment',
-    'version': '0.4.0',
+    'version': '0.4.1',
     'description': 'Integration Test Docker Environment for Exasol',
     'long_description': '# Integration Test Docker Environment\n\n## About\n\nThis project provides a command line interface to start a test environment with an [Exasol Docker-DB](https://hub.docker.com/r/exasol/docker-db). It starts an Exasol Docker-DB container and an associated test container where [EXAPlus CLI](https://docs.exasol.com/connect_exasol/sql_clients/exaplus_cli/exaplus_cli.htm) and  the [Exasol ODBC driver](https://docs.exasol.com/connect_exasol/drivers/odbc.htm) are already installed.\n\nBoth containers exist in the same Docker network. This allows you to connect from the test container to the Docker-DB container. Furthermore, the database gets populated with some test data. You can find the test data under `tests/test/enginedb_small/import.sql`. Besides the test container, you can also access the Exasol database and the Bucket-FS from the host via forwarded ports.\n\n### Comparison to Docker-DB and the Exasol Testcontainers\n\n* This project uses the [Docker-DB](https://hub.docker.com/r/exasol/docker-db), but it does the configuration, setup and startup for you. For example, it waits until the Database and the Bucket-FS accept connections. It also populates the database with test data and provides with the test container as an environment to access the database.\n* The [Exasol Testcontainers](https://github.com/exasol/exasol-testcontainers/) provide similar features for Java integration tests, so if you have an Java project use the Exasol Testcontainer, because they are more integrated in the Java ecosystem then this project. Testcontainers are designed to integrate with popular unit testing frameworks like [JUnit4](https://www.testcontainers.org/test_framework_integration/junit_4/), [JUnit5](https://www.testcontainers.org/test_framework_integration/junit_5/) and [Spock](https://www.testcontainers.org/test_framework_integration/spock/). Most notably, lifecycle management of containers controlled by the tests lifecycle.\n\nHowever, if you have a project in any other language you can use this project to start a test Exasol database via a command line interface.\n\n## Prerequisites\n\nIn order to start a Docker-DB Test Environment, you need:\n\n* Tested Operating System:\n  * Linux\n  * Mac OS X with [Docker Desktop on Mac](https://docs.docker.com/docker-for-mac/install/) and Intel processor\n  * Windows is currently **not supported**\n* Docker with privileged mode\n* At least 2 GiB RAM\n* We recommend at least 15 GB free disk space on the partition \n  where Docker stores its images and containers. On Linux Docker typically stores \n  the images under `/var/lib/docker`.\n\n## Getting started\n\nClone the repository\n\n```\ngit clone https://github.com/exasol/integration-test-docker-environment\n```\n\nStarting the test environment:\n\n```\n./start-test-env spawn-test-environment --environment-name <NAME>\n```\nor if you work on the code of the Test Environment (requires Python >=3.6 with [poetry](https://python-poetry.org/)):\n\n```\n./start-test-env-with-poetry spawn-test-environment --environment-name <NAME>\n```\n\nShutdown of the test environment is currently done manual.\n\n### Options\n\nThe following options are available to customize the test environment. \n\n```\nUsage: main.py spawn-test-environment [OPTIONS]\n\n  This command spawn a test environment with a docker-db container and a\n  connected test-container. The test-container is reachable by the database\n  for output redirects of UDFs.\n\nOptions:\n  --environment-name TEXT         Name of the docker environment. This name\n                                  gets used as suffix for the container\n                                  db_container_<name> and\n                                  test_container_<name>  [required]\n\n  --database-port-forward INTEGER\n                                  Host port to which the database port gets\n                                  forwarded\n\n  --bucketfs-port-forward INTEGER\n                                  Host port to which the BucketFS port gets\n                                  forwarded\n\n  --db-mem-size TEXT              The main memory used by the database. Format\n                                  <number> <unit>, e.g. 1 GiB. The minimum\n                                  size is 1 GB, below that the database will\n                                  not start.  [default: 2 GiB]\n\n  --db-disk-size TEXT             The disk size available for the database.\n                                  Format <number> <unit>, e.g. 1 GiB. The\n                                  minimum size is 100 MiB. However, the setup\n                                  creates volume files with at least 2 GB\n                                  larger size, because the database needs at\n                                  least so much more disk.  [default: 2 GiB]\n\n  --nameserver TEXT               Add a nameserver to the list of DNS\n                                  nameservers which the docker-db should use\n                                  for resolving domain names. You can repeat\n                                  this option to add further nameservers.\n\n  --deactivate-database-setup / --no-deactivate-database-setup\n                                  Deactivates the setup of the spawned\n                                  database, this means no data get populated\n                                  and no JDBC drivers get uploaded. This can\n                                  be used either to save time or as a\n                                  workaround for MacOSX where the\n                                  test_container seems not to be able to\n                                  access the tests directory  [default: False]\n\n  --docker-runtime TEXT           The docker runtime used to start all\n                                  containers\n\n  --docker-db-image-version TEXT  Docker DB Image Version against which the\n                                  tests should run.  [default: 7.0.3]\n\n  --docker-db-image-name TEXT     Docker DB Image Name against which the tests\n                                  should run.  [default: exasol/docker-db]\n\n  --output-directory DIRECTORY    Output directory where the system stores all\n                                  output and log files.  [default:\n                                  .build_output]\n\n  --temporary-base-directory DIRECTORY\n                                  Directory where the system creates temporary\n                                  directories.  [default: /tmp]\n```\n\nYou can look at them on the commandline with:\n\n```\n./start-test-env spawn-test-environment --help \n```\n\n### Default Credentials\n\nThe default credentials for the database are\n\n  * User: `sys`\n  * Password: `exasol`\n  \nand for the Bucket-FS:\n\n  * User: `w`\n  * Password: `write`\n  \nor\n\n  * User: `r`\n  * Password: `read`\n\n### Accessing the Environment Information\n\nThe python setup script creates configuration files on the host and in the test container.\n\nOn the host the container information get stored in the build output directory usually under `.build_output/cache/<ENVIRONMENT_NAME>/*`. In the test container the config file is stored at the root directory `/`.\n\nThe following config files are available:\n\n- environment_info.sh: This file is meant to be source by the bash and then provides the information as environment variables. Here an example for the content:\n\n  ```\n  export ENVIRONMENT_NAME=test\n  export ENVIRONMENT_TYPE=EnvironmentType.docker_db\n\n  # Database IP in environment docker network\n  export ENVIRONMENT_DATABASE_HOST=172.21.0.2\n  export ENVIRONMENT_DATABASE_DB_PORT=8888\n  export ENVIRONMENT_DATABASE_BUCKETFS_PORT=6583\n  export ENVIRONMENT_DATABASE_CONTAINER_NAME=db_container_test\n  export ENVIRONMENT_DATABASE_CONTAINER_NETWORK_ALIASES="exasol_test_database db_container_test"\n  # Database IP in the environment docker network\n  export ENVIRONMENT_DATABASE_CONTAINER_IP_ADDRESS=172.21.0.2\n  export ENVIRONMENT_DATABASE_CONTAINER_VOLUMNE_NAME=db_container_test_volume\n  # Database IP on the docker default bridge which under Linux available from the host\n  export ENVIRONMENT_DATABASE_CONTAINER_DEFAULT_BRIDGE_IP_ADDRESS=172.17.0.3\n\n  export ENVIRONMENT_TEST_CONTAINER_NAME=test_container_test\n  export ENVIRONMENT_TEST_CONTAINER_NETWORK_ALIASES="test_container test_container_test"\n  # Test Container IP in the environment docker network\n  export ENVIRONMENT_TEST_CONTAINER_IP_ADDRESS=172.21.0.3\n  ```\n- environment_info.json: Contains the EnvironmentInfo objects pickled with JsonPickle\n  \n### Currently supported Exasol Versions\n\n* **6.2**: up to 6.2.15\n* **7.0**: up to 7.0.11\n\n\nIf you need further versions, please open an issue.\n\n### Tested Docker Runtimes\n\n- Docker Default Runtime\n- [NVIDIA Container Runtime](https://github.com/NVIDIA/nvidia-container-runtime) for GPU accelerated UDFs\n\n## Mac OS X Support\n\n### What do I need to do to start the Test Environment with Mac OS X\n\nThe Exasol Docker-DB needs per default a bit more than 2 GB of RAM, however the Docker VM on Mac OS X provides often not enough RAM to accommodate this. You should increase the RAM of the Docker VM to at least 4.25 GB or reduce the DB Mem Size for the Exasol Docker-DB to less than 2 GB with `--db-mem-size 1 GiB`.\n\n### What happens under the hood\n\nMac OS X with Docker Desktop for Mac uses a lightweight virtual machine with linux in which the docker daemon runs and the containers get started. This makes [networking](https://docs.docker.com/docker-for-mac/networking/) and [shared directories](https://docs.docker.com/docker-for-mac/osxfs/) more complicated then on Linux.\n\nWe start the python setup script for the test environment in its own Docker container, lets call it `docker runner`, because the library [Luigi](https://luigi.readthedocs.io/en/stable/) can have problems with Mac OS X and to avoid the installation of further dependencies. To support Mac OS X, the `start-test-env` script starts the `docker runner` container and mounts the docker socket at `/var/run/docker.sock` and the directory of the test environment from the Mac OS X host to the container. Then, it starts `start-test-env-without-docker` which then starts the python script. It is important, that the repository gets cloned to the Mac OS X host and not to a docker container, because the python scripts tries to start further docker container which use host mounts to share the tests directory of the test environment with the docker container.\n',
     'author': 'Torsten Kilias',


Test patch:
diff --git a/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py b/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py
index 5029770ea..38c46b806 100644
--- a/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py
+++ b/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py
@@ -132,12 +132,12 @@ def collect_files_and_directories(
             if self.hash_directory_names:
                 new_directories = [d for d in dirs if not self.is_excluded_directory(d)]
                 tmp_collected_directories.extend([(directory, os.path.join(root, d)) for d in new_directories])
-                numCharacters += sum([len(d[1]) for d in new_directories])
+                numCharacters += sum([len(d) for d in new_directories])
 
             new_files = [f for f in files
                          if not self.is_excluded_file(f) and not self.has_excluded_extension(f)]
             tmp_collected_files.extend([(directory, os.path.join(root, f)) for f in new_files])
-            numCharacters += sum([len(f[1]) for f in new_files])
+            numCharacters += sum([len(f) for f in new_files])
 
             if numCharacters > self.MAX_CHARACTERS_PATHS:
                 raise OSError(f"Walking through too many directories. Aborting. Please verify: {directory}")
diff --git a/exasol_integration_test_docker_environment/test/test_hash_temp_dir.py b/exasol_integration_test_docker_environment/test/test_hash_temp_dir.py
index 855787785..76d20f158 100644
--- a/exasol_integration_test_docker_environment/test/test_hash_temp_dir.py
+++ b/exasol_integration_test_docker_environment/test/test_hash_temp_dir.py
@@ -11,15 +11,23 @@
 
 
 class HashTempDirTest(unittest.TestCase):
-    def setUp(self):
+    @classmethod
+    def setUpClass(self):
         self.temp_dir = tempfile.mkdtemp() + "/" + self.__class__.__name__
         self.test_dir1 = self.temp_dir + "/test1"
         self.generate_test_dir(self.test_dir1)
         self.test_dir2 = self.temp_dir + "/test2"
         self.generate_test_dir(self.test_dir2)
+        
+        self.test_dir3 = self.temp_dir + "/test3"
+        os.makedirs(self.test_dir3+"/d") 
+        with open(self.test_dir3+"/f", "wt") as f:
+            f.write("test")
+        
         with open(TEST_FILE, "wt") as f:
             f.write("test")
 
+    @classmethod
     def generate_test_dir(self, test_dir):
         level1 = 5
         level2 = 5
@@ -39,10 +47,23 @@ def generate_test_dir(self, test_dir):
                             with open(test_dir + file, mode="wt") as f:
                                 f.write(file)
 
-    def tearDown(self):
+    @classmethod
+    def tearDownClass(self):
         os.remove(TEST_FILE)
         shutil.rmtree(self.temp_dir)
 
+    def test_single_character_directory_name(self):
+        hasher = FileDirectoryListHasher(hashfunc="sha256",
+                                         use_relative_paths=False,
+                                         hash_directory_names=True,
+                                         hash_file_names=True)
+        old_pwd = os.getcwd()
+        os.chdir(self.test_dir3)
+        hash = hasher.hash(["."])
+        ascii_hash = base64.b32encode(hash).decode("ASCII")
+        self.assertEqual("W5VNU3LMI7OM2ZM7CNBW52YQ7FWZRCW5Z7PHTEUP3KSNOF3OPZXQ====", ascii_hash)
+        os.chdir(old_pwd)
+
     def test_file_content_only_fixed_hash(self):
         hasher = FileDirectoryListHasher(hashfunc="sha256",
                                          use_relative_paths=False,