Repository: vaexio/vaex

Issue:
[BUG-REPORT] Vaex uses more threads than it should / documentation is incomplete
**Description**
The page https://vaex.readthedocs.io/en/latest/conf.html#thread-count specifies that by setting vaex.settings.main.thread_count, you set the thread count, while this in practice this happens only when setting the environment variable VAEX_NUM_THREADS.

This was observed during run on server that had large differences between actual number of processors (96) and number of threads allowed for the particular vaex program (via a batch system) which resulted in the process being killed, even though the program was setting vaex.settings.main.thread_count to a small number of threads.

This seems indeed to be the case by checking the code, thread_count_default is set at https://github.com/vaexio/vaex/blob/98b9f7924d57bf5cc75212182bf0d283d7ebc059/packages/vaex-core/vaex/multithreading.py#L21 but as far as I can tell that does not check at all the settings.

**Software information**
 - Vaex version (`import vaex; vaex.__version__)`: {'vaex': '4.12.0', 'vaex-core': '4.12.0', 'vaex-viz': '0.5.3', 'vaex-hdf5': '0.12.3', 'vaex-server': '0.8.1', 'vaex-astro': '0.9.1', 'vaex-jupyter': '0.8.0', 'vaex-ml': '0.18.0'}
 - Vaex was installed via: pip 
 - OS: Arch (similar behavior on Ubuntu)

**Additional information**
Code to reproduce:
``` 
import vaex
vaex.settings.main.thread_count = 2
vaex.settings.main.thread_count_io = 2

import vaex.dataframe
df = vaex.example()
print(f"vaex.dataframe.main_executor.thread_pool.nthreads={vaex.dataframe.main_executor.thread_pool.nthreads}")
```
results in (incorrect)
```
vaex.dataframe.main_executor.thread_pool.nthreads=12
```
While if you run
```
VAEX_NUM_THREADS=2 python3 test.py
```
you get (the correct):
```
vaex.dataframe.main_executor.thread_pool.nthreads=2
```


Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/csv.py b/packages/vaex-core/vaex/csv.py
--- a/packages/vaex-core/vaex/csv.py
+++ b/packages/vaex-core/vaex/csv.py
@@ -9,10 +9,11 @@
 from frozendict import frozendict
 
 import vaex.dataset
+import vaex.settings
 import vaex.file
 from vaex.dataset import Dataset, DatasetFile
 from .itertools import pmap, pwait, buffer, consume, filter_none
-from .multithreading import thread_count_default_io, get_main_io_pool
+from .multithreading import get_main_io_pool
 
 
 MB = 1024**2
@@ -406,7 +407,7 @@ def chunk_iterator(self, columns, chunk_size=None, reverse=False, start=0, end=N
         for column in columns:
             if column not in self._columns:
                 self._check_existence(column)
-        for c1, c2, chunks in filter_none(pwait(buffer(chunk_generator, thread_count_default_io//2+3))):
+        for c1, c2, chunks in filter_none(pwait(buffer(chunk_generator, vaex.settings.main.thread_count_io//2+3))):
             chunks_ready_list.append(chunks)
             total_row_count = sum([len(list(k.values())[0]) for k in chunks_ready_list])
             if total_row_count > chunk_size:
diff --git a/packages/vaex-core/vaex/multithreading.py b/packages/vaex-core/vaex/multithreading.py
--- a/packages/vaex-core/vaex/multithreading.py
+++ b/packages/vaex-core/vaex/multithreading.py
@@ -2,24 +2,18 @@
 from __future__ import division, print_function
 import asyncio
 import threading
-import queue
-import math
 import multiprocessing
-import sys
+from warnings import warn
 import vaex.utils
 import logging
-import traceback
-import cProfile
 import concurrent.futures
 import time
-import os
+import vaex.settings
 
 from .itertools import buffer
 
 logger = logging.getLogger("vaex.multithreading")
 
-thread_count_default = vaex.utils.get_env_type(int, 'VAEX_NUM_THREADS', multiprocessing.cpu_count())
-thread_count_default_io = vaex.utils.get_env_type(int, 'VAEX_NUM_THREADS_IO', thread_count_default + 1)
 main_pool = None
 main_io_pool = None
 
@@ -44,7 +38,7 @@ def get_main_pool():
 def get_main_io_pool():
     global main_io_pool
     if main_io_pool is None:
-        main_io_pool = concurrent.futures.ThreadPoolExecutor(max_workers=thread_count_default_io)
+        main_io_pool = concurrent.futures.ThreadPoolExecutor(max_workers=vaex.settings.main.thread_count_io)
     return main_io_pool
 
 
@@ -59,7 +53,7 @@ class ThreadPoolIndex(concurrent.futures.ThreadPoolExecutor):
 
     def __init__(self, max_workers=None, *args, **kwargs):
         if max_workers is None:
-            max_workers = thread_count_default
+            max_workers = vaex.settings.main.thread_count
         super(ThreadPoolIndex, self).__init__(max_workers, *args, **kwargs)
         self.lock = threading.Lock()
         self.thread_indices = iter(range(1000000))  # enough threads until 2100?


Test patch: