Repository: log2timeline/plaso

Issue #1970: OverflowError: Python int too large to convert to SQLite INTEGER
Created: 2018-06-22T06:26:01Z
Description of problem:
log2timeline crashes while analyzing a Firefox's cookies.sqlite file with Python exception OverflowError: Python int too large to convert to SQLite INTEGER.
Command line and arguments:
log2timeline --status_view linear /path/file.plaso cookies.sqlite

Source data:
This data is sensitive, I can't provide it unfortunately.
Plaso version:
20180524
Operating system Plaso is running on:
Plaso is running inside a Docker container based on an Ubuntu 16.04 image.
Installation method:
Relevant part of the Dockerfile:
RUN add-apt-repository -y "deb http://ppa.launchpad.net/gift/stable/ubuntu xenial main"
RUN apt-get update && apt-get -y upgrade && apt-get -y dist-upgrade
RUN apt-get -y install python-plaso plaso-tools

Debug output/tracebacks:
$ log2timeline -d --status_view linear  /path/file.plaso /path/cookies.sqlite
2018-06-22 06:13:55,562 [WARNING] (MainProcess) PID:7 <tool_options> Appending to an already existing storage file.
2018-06-22 06:13:55,749 [DEBUG] (MainProcess) PID:7 <log2timeline_tool> Starting extraction in single process mode.
2018-06-22 06:13:55,768 [DEBUG] (MainProcess) PID:7 <extractors> Active parsers: amcache, android_app_usage, asl_log, bash, bencode, binary_cookies, bsm_log, chrome_cache, chrome_preferences, cups_ipp, custom_destinations, dockerjson, dpkg, esedb, filestat, firefox_cache, firefox_cache2, fsevents, gdrive_synclog, hachoir, java_idx, lnk, mac_appfirewall_log, mac_keychain, mac_securityd, mactime, macwifi, mcafee_protection, mft, msiecf, olecf, openxml, opera_global, opera_typed_history, pe, plist, pls_recall, popularity_contest, prefetch, recycle_bin, recycle_bin_info2, rplog, sccm, selinux, skydrive_log, skydrive_log_old, sophos_av, sqlite, symantec_scanlog, syslog, systemd_journal, trendmicro_vd, usnjrnl, utmp, utmpx, winevt, winevtx, winfirewall, winiis, winjob, winreg, xchatlog, xchatscrollback, zsh_extended_history
2018-06-22 06:13:55,768 [DEBUG] (MainProcess) PID:7 <hashing_analyzer> Got hasher names: sha256
2018-06-22 06:13:55,768 [DEBUG] (MainProcess) PID:7 <single_process> Processing started.
2018-06-22 06:13:55,772 [DEBUG] (MainProcess) PID:7 <worker> [ProcessFileEntry] processing file entry: OS:/path/cookies.sqlite
2018-06-22 06:13:55,772 [DEBUG] (MainProcess) PID:7 <worker> [ProcessFileEntryDataStream] processing data stream: "" of file entry: OS:/path/cookies.sqlite
2018-06-22 06:13:55,772 [DEBUG] (MainProcess) PID:7 <worker> [AnalyzeDataStream] analyzing file: OS:/path/cookies.sqlite
2018-06-22 06:13:55,778 [DEBUG] (MainProcess) PID:7 <hashing_analyzer> Processing results for hasher sha256
2018-06-22 06:13:55,778 [DEBUG] (MainProcess) PID:7 <worker> [AnalyzeFileObject] attribute sha256_hash:379ac9073895c45ce81540d6e264666ef40918b52627bac169325b9d6b591f31 calculated for file: OS:/path/cookies.sqlite
2018-06-22 06:13:55,778 [DEBUG] (MainProcess) PID:7 <worker> [AnalyzeDataStream] completed analyzing file: OS:/path/cookies.sqlite
2018-06-22 06:13:55,778 [DEBUG] (MainProcess) PID:7 <worker> [ExtractMetadataFromFileEntry] processing file entry: OS:/path/cookies.sqlite
2018-06-22 06:13:55,781 [DEBUG] (MainProcess) PID:7 <extractors> [ParseFileEntryWithParsers] parsing file: OS:/path/cookies.sqlite with parser: sqlite
2018-06-22 06:13:57,935 [DEBUG] (MainProcess) PID:7 <worker> [ProcessFileEntry] done processing file entry: OS:/path/cookies.sqlite
Checking availability and versions of dependencies.
[OPTIONAL]      missing: lzma.
[OK]


Source path     : /path/cookies.sqlite
Source type     : single file
Processing time : 00:00:00

Processing started.
Traceback (most recent call last):
  File "/usr/bin/log2timeline.py", line 68, in <module>
    if not Main():
  File "/usr/bin/log2timeline.py", line 54, in Main
    tool.ExtractEventsFromSources()
  File "/usr/lib/python2.7/dist-packages/plaso/cli/log2timeline_tool.py", line 433, in ExtractEventsFromSources
    status_update_callback=status_update_callback)
  File "/usr/lib/python2.7/dist-packages/plaso/engine/single_process.py", line 283, in ProcessSources
    storage_writer.Close()
  File "/usr/lib/python2.7/dist-packages/plaso/storage/interface.py", line 1308, in Close
    self._storage_file.Close()
  File "/usr/lib/python2.7/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 665, in Close
    self._WriteSerializedAttributeContainerList(self._CONTAINER_TYPE_EVENT)
  File "/usr/lib/python2.7/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 445, in _WriteSerializedAttributeContainerList
    self._cursor.executemany(query, values_tuple_list)
OverflowError: Python int too large to convert to SQLite INTEGER

Duplicate issue #2055: Hang/stall on GZIP files
Created: 2018-07-24T17:49:14Z
Description of problem:
log2timeline hangs on the following specific Gzip files. The following screen output doesn't change at all after running for over an hour:
plaso - log2timeline version 20180127

Source path	: /dev/xvdf
Source type	: storage media device

Tasks:          Queued  Processing      To merge        Abandoned       Total
                0       7               0               3               61659

Identifier      PID     Status          Memory          Sources         Events          File
Main            21688   running         607.9 MiB       61656 (0)       240505 (0)      
Worker_00       21695   idle            269.7 MiB       4143 (0)        15438 (0)       GZIP:/usr/share/doc/cryptsetup/FAQ.gz
Worker_01       21699   aborted         276.5 MiB       4816 (0)        15394 (0)       GZIP:/usr/share/doc/linux-image-4.4.0-1061-aws/changelog.Debian.gz
Worker_02       21703   idle            268.3 MiB       3464 (0)        16879 (0)       GZIP:/usr/share/man/man1/pl2pm.1.gz
Worker_03       21707   aborted         276.5 MiB       2961 (0)        15362 (0)       GZIP:/usr/share/doc/linux-aws-headers-4.4.0-1061/changelog.Debian.gz
Worker_04       21711   idle            263.8 MiB       3161 (0)        15429 (0)       GZIP:/usr/share/info/screen.info-4.gz
Worker_05       21715   idle            283.5 MiB       4005 (0)        18305 (0)       GZIP:/usr/share/man/fr/man8/vigr.8.gz
Worker_06       21719   aborted         284.9 MiB       4079 (0)        9965 (48)       GZIP:/usr/share/doc/linux-aws-headers-4.4.0-1061/changelog.Debian.gz
Worker_07       21723   idle            278.9 MiB       4440 (0)        15407 (0)       GZIP:/usr/share/man/es/man1/whatis.1.gz
Worker_08       21727   idle            280.6 MiB       4482 (0)        15902 (0)       GZIP:/usr/share/man/man5/nanorc.5.gz
Worker_09       21731   aborted         272.4 MiB       6200 (0)        9855 (44)       GZIP:/usr/share/doc/linux-image-4.4.0-1061-aws/changelog.Debian.gz
Worker_10       21735   idle            272.1 MiB       5304 (0)        15525 (0)       GZIP:/usr/share/info/gzip.info.gz
Worker_11       21739   aborted         273.9 MiB       3292 (0)        15426 (0)       GZIP:/usr/share/doc/linux-headers-4.4.0-1061-aws/changelog.Debian.gz
Worker_12       21743   idle            276.0 MiB       3843 (0)        15403 (0)       GZIP:/usr/share/man/pt/man1/debconf-escape.1.gz
Worker_13       21747   idle            275.4 MiB       3576 (0)        15318 (0)       GZIP:/usr/share/man/man7/iso_8859-3.7.gz
Worker_14       21751   idle            283.6 MiB       3170 (0)        15382 (0)       GZIP:/usr/share/man/de/man8/fsck.fat.8.gz
Worker_15       21756   idle            549.5 MiB       0 (0)           0 (0)           
Worker_16       21760   idle            549.3 MiB       0 (0)           0 (0)           
Worker_17       21764   idle            549.3 MiB       0 (0)           0 (0)           
Worker_18       21768   aborted         552.7 MiB       2 (0)           1534 (40)       GZIP:/usr/share/doc/linux-headers-4.4.0-1061-aws/changelog.Debian.gz
Worker_19       21772   idle            550.5 MiB       0 (0)           0 (0)           
Worker_20       21776   idle            552.9 MiB       186 (0)         4888 (0)        GZIP:/usr/share/doc/libgnutls30/THANKS.gz
Worker_21       21780   idle            563.3 MiB       172 (0)         2977 (0)        GZIP:/usr/share/man/man5/update-motd.5.gz
Worker_22       21784   idle            563.3 MiB       195 (0)         3039 (0)        GZIP:/usr/share/man/fr/man1/manpath.1.gz
Worker_23       21788   idle            563.3 MiB       164 (0)         3000 (0)        GZIP:/usr/share/man/ru/man8/deluser.8.gz
Worker_24       21835   idle            588.7 MiB       0 (0)           0 (0)           
Worker_25       21839   idle            589.2 MiB       0 (0)           0 (0)           
Worker_26       21843   idle            589.2 MiB       0 (0)           0 (0)           
Worker_27       21847   idle            588.9 MiB       0 (0)           0 (0)           
Worker_28       21851   idle            589.2 MiB       0 (0)           0 (0)           
Worker_29       21855   idle            589.2 MiB       0 (0)           0 (0)           
Worker_30       21859   idle            589.2 MiB       0 (0)           0 (0)           
Worker_31       21863   idle            589.2 MiB       0 (0)           0 (0)           
Worker_32       21867   idle            589.4 MiB       0 (0)           0 (0)           

Command line and arguments:
sudo log2timeline.py -z UTC --logfile l2ttest.plaso.log.gz --debug lt2test.plaso /dev/xvdf
If I include the --skip_compressed_streams flag, the problem does not occur and the processing completes as expected.
Source data:
The above command line was ran against an 8GB AWS EBS volume created from the root device of an AWS EC2 instance that was running Ubuntu Server 16.04.
Plaso version:
20180127
Ubuntu Server 16.04.4 LTS
Kernel 4.4.0-1055-aws
Installation method:
Installed along with other SANS SIFT packages using the sift-cli tool with --packages-only mode. Running on an AWS EC2 instance (instance type c4.4xlarge)
Debug output/tracebacks:
I didn't see anything in the debug log that was meaningful to me, though I'm not sure what I should be looking for. Here are all the WARNING and ERROR messages grep'd out, with 2 lines of context:
2018-07-24 17:22:56,310 [DEBUG] (MainProcess) PID:21688 <extraction_tool> Starting preprocessing.
2018-07-24 17:22:56,353 [WARNING] (MainProcess) PID:21688 <manager> Unable to collect value from artifact definition: LinuxSystemdOSRelease with error: Not a regular file.
2018-07-24 17:22:56,527 [INFO] (MainProcess) PID:21688 <engine> Preprocessing detected operating systems: Linux
2018-07-24 17:22:56,527 [DEBUG] (MainProcess) PID:21688 <extraction_tool> Preprocessing done.
--
2018-07-24 17:25:19,736 [DEBUG] (MainProcess) PID:21688 <task_manager> Task 7b45472c6e8548e28fe9813ccf841ea5 was queued, now processing
2018-07-24 17:25:19,736 [DEBUG] (MainProcess) PID:21688 <zeromq_queue> main_task_queue sending item
2018-07-24 17:25:19,737 [ERROR] (MainProcess) PID:21688 <engine> Process Worker_09 (PID: 21731) is not functioning correctly. Status code: aborted.
2018-07-24 17:25:19,737 [DEBUG] (MainProcess) PID:21688 <zeromq_queue> main_task_queue sent item
2018-07-24 17:25:19,738 [WARNING] (MainProcess) PID:21688 <engine> Terminating process: (PID: 21731).
2018-07-24 17:25:19,739 [DEBUG] (MainProcess) PID:21688 <task_manager> Task 64ee8e237a464bd6815bf59ea13c98d6 is complete.
2018-07-24 17:25:19,740 [DEBUG] (MainProcess) PID:21688 <task_engine> Scheduled task fd3483d694874082b4980a733227cbb3 for path specification type: OS, location: /dev/xvdf
--
2018-07-24 17:25:24,149 [DEBUG] (MainProcess) PID:21688 <task_manager> Task eff32caf8b654b29a969d0635a54585a is pending merge
2018-07-24 17:25:24,150 [DEBUG] (MainProcess) PID:21688 <task_manager> Task 6208117629204ba7a9417245360e2f89 was queued, now processing
2018-07-24 17:25:24,151 [ERROR] (MainProcess) PID:21688 <engine> Process Worker_06 (PID: 21719) is not functioning correctly. Status code: aborted.
2018-07-24 17:25:24,151 [WARNING] (MainProcess) PID:21688 <engine> Terminating process: (PID: 21719).
2018-07-24 17:25:24,152 [DEBUG] (MainProcess) PID:21688 <task_manager> Task 37da7edaa0cc4c9cb86881aa8d39c852 is complete.
2018-07-24 17:25:24,153 [DEBUG] (MainProcess) PID:21688 <task_engine> Scheduled task 055eff18485348a587919d921fbe26dd for path specification type: OS, location: /dev/xvdf
--

2018-07-24 17:25:45,473 [DEBUG] (MainProcess) PID:21688 <task_manager> Task 9195bec48d684e58bec7a2598c2ce00d was queued, now processing
2018-07-24 17:25:45,473 [ERROR] (MainProcess) PID:21688 <engine> Process Worker_18 (PID: 21768) is not functioning correctly. Status code: aborted.
2018-07-24 17:25:45,473 [DEBUG] (MainProcess) PID:21688 <task_manager> Task 07b102ffeb7942f89e80057a16cabc38 is pending merge
2018-07-24 17:25:45,473 [WARNING] (MainProcess) PID:21688 <engine> Terminating process: (PID: 21768).
2018-07-24 17:25:45,473 [DEBUG] (MainProcess) PID:21688 <task_manager> Task 4f2a75c4d2c4449c9f3a83912be848a5 is pending merge
2018-07-24 17:25:45,474 [DEBUG] (MainProcess) PID:21688 <zeromq_queue> main_task_queue sending item
--
2018-07-24 17:30:19,741 [DEBUG] (MainProcess) PID:21688 <zeromq_queue> main_task_queue sent item
2018-07-24 17:30:19,862 [DEBUG] (MainProcess) PID:21688 <task_manager> Task a13d3d5cc1f245e78c431d922520c41e was queued, now processing
2018-07-24 17:30:19,862 [ERROR] (MainProcess) PID:21688 <engine> Process Worker_01 (PID: 21699) is not functioning correctly. Status code: aborted.
2018-07-24 17:30:19,863 [WARNING] (MainProcess) PID:21688 <engine> Terminating process: (PID: 21699).
2018-07-24 17:30:23,387 [DEBUG] (MainProcess) PID:21688 <engine> Stopped monitoring process: Worker_01 (PID: 21699)
2018-07-24 17:30:23,388 [INFO] (MainProcess) PID:21688 <engine> Starting replacement worker process for Worker_01
--
2018-07-24 17:30:24,158 [DEBUG] (MainProcess) PID:21688 <zeromq_queue> main_task_queue sent item
2018-07-24 17:30:24,254 [DEBUG] (MainProcess) PID:21688 <task_manager> Task f8d264771d424e41a44cb5e4633d5988 was queued, now processing
2018-07-24 17:30:24,254 [ERROR] (MainProcess) PID:21688 <engine> Process Worker_03 (PID: 21707) is not functioning correctly. Status code: aborted.
2018-07-24 17:30:24,254 [WARNING] (MainProcess) PID:21688 <engine> Terminating process: (PID: 21707).
2018-07-24 17:30:27,779 [DEBUG] (MainProcess) PID:21688 <engine> Stopped monitoring process: Worker_03 (PID: 21707)
2018-07-24 17:30:27,780 [INFO] (MainProcess) PID:21688 <engine> Starting replacement worker process for Worker_03
--
2018-07-24 17:30:45,479 [DEBUG] (MainProcess) PID:21688 <zeromq_queue> main_task_queue sent item
2018-07-24 17:30:45,907 [DEBUG] (MainProcess) PID:21688 <task_manager> Task b6856efb93554a8a8a87cc7853b5d9ad was queued, now processing
2018-07-24 17:30:45,907 [ERROR] (MainProcess) PID:21688 <engine> Process Worker_11 (PID: 21739) is not functioning correctly. Status code: aborted.
2018-07-24 17:30:45,907 [WARNING] (MainProcess) PID:21688 <engine> Terminating process: (PID: 21739).
2018-07-24 17:30:49,432 [DEBUG] (MainProcess) PID:21688 <engine> Stopped monitoring process: Worker_11 (PID: 21739)
2018-07-24 17:30:49,433 [INFO] (MainProcess) PID:21688 <engine> Starting replacement worker process for Worker_11

Closing comment (2018-11-07T22:56:17Z): This looks like a duplicate of #1970 which was addressed in Plaso 20180703. Closing.