Repository: log2timeline/plaso

Issue #3164: Unable to run psteal 20200717 in vitualenv on MacOS Catalina
Created: 2020-07-21T14:10:40Z
Description of problem:
Hello,
I'm unable to use psteal with the last version of the repository, I have an error about multiprocessing
Thank you for your help
Command line and arguments:
/usr/local/bin/psteal.py --source /Users/user/Desktop/timesketch/art --additional_fields "data_type, strings, event_type" -w /Users/user/Desktop/timesketch/extract.csv
Source data:
This is a CyLR output, I can't share it sorry
Plaso version:
plaso 20200717 (git clone on it)
Operating system Plaso is running on:
MacOS 10.15.6 (Catalina)
Installation method:
In a virtualenv;
pip3 install -r requirements.txt
python3 setup.py install

Debug output/tracebacks:
2020-07-21 16:03:50,352 [INFO] (MainProcess) PID:4045 <data_location> Determined data location: /usr/local/lib/python3.8/site-packages/plaso-20200717-py3.8.egg/share/plaso
2020-07-21 16:03:50,545 [INFO] (MainProcess) PID:4045 <artifact_definitions> Determined artifact definitions path: /Users/user/Desktop/Files/Tools/plaso/plaso/share/artifacts
Checking availability and versions of dependencies.
[OK]


Source path     : /Users/user/Desktop/timesketch/art
Source type     : directory
Processing time     : 00:00:00

Processing started.
Traceback (most recent call last):
  File "/usr/local/bin/psteal.py", line 4, in <module>
    __import__('pkg_resources').run_script('plaso==20200717', 'psteal.py')
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 667, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1464, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.8/site-packages/plaso-20200717-py3.8.egg/EGG-INFO/scripts/psteal.py", line 110, in <module>
    if not Main():
  File "/usr/local/lib/python3.8/site-packages/plaso-20200717-py3.8.egg/EGG-INFO/scripts/psteal.py", line 86, in Main
    tool.ExtractEventsFromSources()
  File "/usr/local/lib/python3.8/site-packages/plaso-20200717-py3.8.egg/plaso/cli/psteal_tool.py", line 388, in ExtractEventsFromSources
    processing_status = extraction_engine.ProcessSources(
  File "/usr/local/lib/python3.8/site-packages/plaso-20200717-py3.8.egg/plaso/multi_processing/task_engine.py", line 817, in ProcessSources
    extraction_process = self._StartWorkerProcess('', storage_writer)
  File "/usr/local/lib/python3.8/site-packages/plaso-20200717-py3.8.egg/plaso/multi_processing/task_engine.py", line 572, in _StartWorkerProcess
    process.start()
  File "/usr/local/Cellar/python@3.8/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/Cellar/python@3.8/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/Cellar/python@3.8/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/Cellar/python@3.8/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Cellar/python@3.8/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/Cellar/python@3.8/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/usr/local/Cellar/python@3.8/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 183, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: module '__main__' has no attribute '__spec__'

Duplicate issue #3286: nsrlsvr analysis plugin reaches worker memory limit
Created: 2020-11-04T05:51:43Z
Description of problem:
Not really new problem, but was reported before?
After numerous tries on various versions of plaso, with latest nsrlsvr (1.7.0 released), the process will stop halfway through.
Noticed the worker module "nsrlsvr", when it hits 2GB in memory usage, the "nsrlsvr" process is killed.
Tried it on at least 3 differing plaso files. Only the plaso which hits less then 2GB of nsrlsvr memory use, seems to be ok.
Command line and arguments:
psort.py --analysis nsrlsvr -o null <plasofile>
Plaso version:
20201007 - installed via GIFT -> plaso-tools
Tried on github version as well. Same results.
Tried on 0730 version. Same results.
Operating system Plaso is running on:
Ubuntu 20.04.1
Clean install on VM
Installation method:
sudo add-apt-repository ppa:gift/stable
sudo apt-get update
sudo apt-get install plaso-tools


Debug output/tracebacks:
Traceback (most recent call last):
  File "/usr/bin/psort.py", line 99, in <module>
    if not Main():
  File "/usr/bin/psort.py", line 76, in Main
    tool.ProcessStorage()
  File "/usr/lib/python3/dist-packages/plaso/cli/psort_tool.py", line 572, in ProcessStorage
    analysis_engine.AnalyzeEvents(
  File "/usr/lib/python3/dist-packages/plaso/multi_processing/psort.py", line 949, in AnalyzeEvents
    self._AnalyzeEvents(
  File "/usr/lib/python3/dist-packages/plaso/multi_processing/psort.py", line 321, in _AnalyzeEvents
    event_queue.PushItem((event, event_data, event_data_stream))
  File "/usr/lib/python3/dist-packages/plaso/engine/zeromq_queue.py", line 457, in PushItem
    raise errors.QueueFull
plaso.lib.errors.QueueFull

Closing comment (2020-12-08T06:00:24Z): @madsumm that is a duplicate of #3164 and looks like an issue with your Python installation.
thx for confirming that the limit is no longer an issue, now to figure out why such a vast amount of memory is being consumed