deardaniel/secdata-raw huggingface dataset:
Log2timeline timeline generation: Analysis plugin sessionize

# Sessionize analysis plugin

Notes on how to use the sessionize analysis plugin.

## Running the analysis plugin

First run log2timeline to extract events:

```bash
log2timeline.py --storage-file timeline.plaso image.raw
```

Next run psort to determine sessions:

```bash
psort.py --analysis sessionize -o null timeline.plaso
```

This will tag with a label to indicate which session they belong to.

The last step would be to export a timeline with the tags:

```bash
psort.py -o dynamic --fields datetime,timestamp_desc,source,source_long,message,parser,tag -w timeline.csv timeline.plaso
```
Log2timeline timeline generation: Analysis plugin tagging

# Tagging Analysis Plugin

Notes on how to use the tagging analysis plugin.

## Creating a tagging file

A tagging file is an UTF-8 encoded text file that contains tagging rules.

A tagging rule consists of:

```
# Short description
LABEL
  EVENT FILTER EXPRESSION
```

For example:

```
task_schedule
  data_type is 'windows:evt:record' and source_name is 'Security' and event_identifier is 602
  data_type is 'windows:evtx:record' and source_name is 'Microsoft-Windows-Security-Auditing' and event_identifier is 4698
```

## Running the analysis plugin

First run log2timeline to extract events:

```bash
log2timeline.py --storage-file timeline.plaso image.raw
```

Next run psort to tag events:

```bash
psort.py --analysis tagging --tagging-file tagging-file.txt -o null timeline.plaso
```

This will tag events based on the tagging rules.

The last step would be to export a timeline with the tags:

```bash
psort.py -o dynamic --fields datetime,timestamp_desc,source,source_long,message,parser,tag -w timeline.csv timeline.plaso
```

## Also see

* [Event filters](Event-filters.md)
* [Tagging rules](Tagging-Rules.md)
Log2timeline timeline generation: Feature requests and bug reports

# Feature requests and bug reports

## I would like to see support for a specific file format

If you like to see support for a specific file format here is how you can help.

1. Collect or create test data that can be publicly shared. Preferable test data that represents the necessary edge cases.
1. Document the data format, including the edge cases and common corruption cases. For example see [dtFormats](https://github.com/libyal/dtformats/tree/main/documentation).
1. Write a parser (or parser plugin) with tests and test data and submit as a PR to the project.

For more context on this approach see: [Testing digital forensic data processing tools](https://osdfir.blogspot.com/2020/09/testing-digital-forensic-data.html).

## GitHub issue tracker

Plaso feature requests and bug reports are tracked on the [GitHub issue tracker](https://github.com/log2timeline/plaso/issues).

### Labels

Feature requests in GitHub are labelled with "enhancement" for Plaso we
additionally label them with one or more of the following "focus area" labels:

* analysis; changes to analysis plug-ins
* core; changes to the core
* deployment; changes to deployment or development utility scripts
* parsers; changes to parsers and parser plug-ins
* testing; changes to improve testing
* UX; changes to user experience

### Milestones

After every release the Plaso maintainers will go over the open feature
requests and bug reports and determine which of the issues we are going to give
priority to work on and marked by a corresponding milestone.

Note that this is no guarantee that the feature will actually be implemented in
the next release. This is depend on other factors e.g. available time, show
stopper bugs, etc.
Log2timeline timeline generation: Analysis plugin viper

# Viper Analysis Plugin

Notes on how to use the viper analysis plugin.

## Setting up Viper

The Viper project maintains installation instructions here:
https://viper-framework.readthedocs.io/en/latest/installation/index.html

## Running the analysis plugin

First run log2timeline to extract events:

```bash
log2timeline.py --storage-file timeline.plaso image.raw
```

Note that hashing must be turned on for the viper plugin to work correctly.
This is default setting for log2timeline.py.

Next run psort to tag events, then output them:

```bash
psort.py --analysis viper -o null timeline.plaso
```

If a file processed by Plaso is present in the viper instance, it will be
tagged with `viper_present`. If it's part of a project in viper, it will also
be tagged with `viper_project_$PROJECTNAME`.

The last step would be to export a timeline with the tags:

```bash
psort.py -o dynamic --fields datetime,timestamp_desc,source,source_long,message,parser,tag -w timeline.csv timeline.plaso
```
Log2timeline timeline generation: Analysis plugin browser search

# Browser search analysis plugin

Notes on how to use the browser_search analysis plugin.

## Running the analysis plugin

First run log2timeline to extract events:

```bash
log2timeline.py --storage-file timeline.plaso image.raw
```

Note that the browser search analysis plugin analyzes URLS in web history data
such as events with data type:

* chrome:autofill:entry
* chrome:cache:entry
* chrome:cookie:entry
* chrome:extension_activity:activity_log
* chrome:history:file_downloaded
* chrome:history:page_visited
* cookie:google:analytics:utma
* cookie:google:analytics:utmb
* cookie:google:analytics:utmt
* cookie:google:analytics:utmz
* firefox:cache:record
* firefox:cookie:entry
* firefox:downloads:download
* firefox:places:bookmark
* firefox:places:bookmark_annotation
* firefox:places:bookmark_folder
* firefox:places:page_visited
* msiecf:leak
* msiecf:redirected
* msiecf:url
* msie:webcache:container
* msie:webcache:containers
* msie:webcache:leak_file
* msie:webcache:partitions
* opera:history:entry
* opera:history:typed_entry
* safari:cookie:entry
* safari:history:visit
* safari:history:visit_sqlite

Next run psort to determine the browser searches:

```bash
psort.py --analysis browser_search -o null timeline.plaso
```

This will:

* extract information of searches performed in a browser, such as search engine, search term and number of queries
* tag corresponding events with the `browser_search` label

The analysis results can be reviewed with pinfo:

```bash
pinfo.py --report browser_search timeline.plaso
```
Log2timeline timeline generation: Analysis plugins

# Analysis Plugins

* [browser_search](Analysis-plugin-browser-search.md): Analyze browser search entries from events.
* [chrome_extension](Analysis-plugin-chrome-extension.md): Analysis plugin to gather information about Chrome extensions.
* [bloom](Analysis-plugin-bloom.md): Analysis plugin for looking up hashes in a bloom file.
* [nsrlsvr](Analysis-plugin-nsrlsvr.md): Analysis plugin for looking up hashes in nsrlsvr.
* [sessionize](Analysis-plugin-sessionize.md): Analysis plugin that labels events by session.
* [tagging](Analysis-plugin-tagging.md): Analysis plugin that labels events according to rules in a tagging file.
* [unique_domains_visited](Analysis-plugin-unique-domains-visited.md): A plugin to generate a list all domains visited.
* [viper](Analysis-plugin-viper.md): An analysis plugin for looking up SHA256 hashes in Viper.
* [virustotal](Analysis-plugin-virustotal.md): An analysis plugin for looking up hashes in VirusTotal.
Log2timeline timeline generation: Fedora Packaged Release

# Fedora Packaged Release

### Fedora 39 and 40

**Note that other versions of Fedora are not supported at this time.**

To install Plaso and dependencies from the GIFT Cool Other Package Repo (COPR)
you'll need to have dnf plugins installed:

```
sudo dnf install dnf-plugins-core
```

Add the [GIFT COPR](https://copr.fedorainfracloud.org/groups/g/gift/coprs/) to
your dnf configuration:

```
sudo dnf copr enable @gift/stable
```

Install Plaso, tools and dependencies:

```
sudo dnf install plaso-tools
```
Log2timeline timeline generation: Tips and Tricks

# Tips and Tricks

This is a collection of few tips and tricks that can be used with **Plaso**

## Import the output of a third party tool into Plaso

If want to import the output of a third party tool into your Plaso timeline
export it to bodyfile format. The Plaso bodyfile parser can parse a bodyfile.

Note that the bodyfile format has numerous limitations see:
[ForensicsWiki: Bodyfile](https://forensics.wiki/bodyfile)

The Plaso bodyfile parser supports timestamps with a fraction of a second since
[Aug 25, 2020](https://github.com/log2timeline/plaso/commit/c81ef3aea9817646ea2846376ce9e2a83c7d5fe5).
Log2timeline timeline generation: Analysis plugin unique domains visited

# Unique domains visited analysis plugin

Notes on how to use the unique_domains_visited analysis plugin.

## Running the analysis plugin

First run log2timeline to extract events:

```bash
log2timeline.py --storage-file timeline.plaso image.raw
```

Note that the unique domains visited analysis plugin analyzes URLS in web
history data such as events with data type:

* chrome:history:file_downloaded
* chrome:history:page_visited
* firefox:downloads:download
* firefox:places:page_visited
* macosx:lsquarantine
* msiecf:redirected
* msiecf:url
* msie:webcache:container
* opera:history
* safari:history:visit

Next run psort to determine the unique domains visited:

```bash
psort.py --analysis unique_domains_visited -o null timeline.plaso
```

This will extract information of unique domains which will be stored in
the analysis report.
Log2timeline timeline generation: index

#######################
Developer documentation
#######################

.. toctree::
   :maxdepth: 3

   Developers Guide <Developers-Guide>
   Style guide <Style-guide>
   Testing <Testing>
   How to write a parser or (parser) plugin <How-to-write-a-parser>
   How to write an analysis plugin <How-to-write-an-analysis-plugin>
   How to write an output module <How-to-write-an-output-module>
   Packaging with Docker <Packaging-with-docker>
Log2timeline timeline generation: Developing Ubuntu

# Developing on Ubuntu

## Git
To download the latest version of Plaso you'll need to install the git tools:
```
sudo apt-get install git
```

Checkout the Plaso source from the git repo:
```
git clone https://github.com/log2timeline/plaso.git
```

## Dependencies

Install the dependencies, using either:
* A [virtualenv](Developing-Virtualenv.html#Ubuntu)
* The [prepackaged dependencies](Development-Dependencies.html#Ubuntu).

Check if you have all the dependencies installed and have the right minimum
versions:
```
python utils/check_dependencies.py
```

**Note that some dependencies are actively under development and can be
frequently updated, therefore we recommend checking the status of the
 dependencies regularly.**

## Updating your environment.

If you are using a github fork your origin is pointing to your fork and not
the main Plaso git repository. When you run `git remote -v` you might see
something like:
```
origin	https://github.com/Onager/plaso (fetch)
origin	https://github.com/Onager/plaso (push)
```

Add a git remote called 'upstream' that you can use to sync your fork:
```
git remote add upstream https://github.com/log2timeline/plaso.git
git pull --rebase upstream main
```

We provide packaged versions of the dependencies via the [l2tbinaries project](https://github.com/log2timeline/l2tbinaries/blob/main/README.md).
However it is possible that the dependencies are not fully up to date therefore
we also provide a build script as part of [l2tdevtools project](https://github.com/log2timeline/l2tdevtools)
to do unattended bulk builds.

## Development tools
If you intend to do development on Plaso you'll also need to install some
development tools:

* PyLint 3.2.x
Log2timeline timeline generation: Developing MacOS

# Developing on MacOS

## XCode

First install XCode

Next install XCode Command Line Tools:

```bash
xcode-select --install
```

## Git

Install the [git tools](https://git-scm.com/downloads).

Checkout the plaso source from the git repo:

```bash
git clone https://github.com/log2timeline/plaso.git
```

## Dependencies

Install the dependencies in a [virtualenv environment](../user/MacOS-Source-Release.html#install-plaso-contained-within-a-virtual-environment)

Check if you have all the dependencies installed and have the right minimum
version:

```bash
./utils/check_dependencies.py
```

**Note that some dependencies are actively under development and can be
frequently updated, therefore we recommend checking the status of the
dependencies regularly.**

## Development tools

If you intend to do development on plaso you'll also need to install some development tools:

* PyLint 3.2.x
Log2timeline timeline generation: Testing

# Testing

Plaso comes with multiple forms of tests:

* unit tests
* end-to-end tests

## Unit tests

The unit tests are intended to detect issues after individual changes (pull
requests). Unit tests are typically small by nature and only test a specific
part of the code.

The unit tests are stored in the `tests` sub directory, in the same parent
directory as the plaso module directory. The unit tests can be run with:

```
python run_tests.py
```

Or if you have tox installed:

```
tox -epy310
```

The unit tests are also run on [GitHub](https://github.com/log2timeline/plaso/actions)
and [AppVeyor](https://ci.appveyor.com/project/log2timeline/plaso) after every
pull request or commit to detect issues early.

## End-to-end tests

The end-to-end tests are intended to test the Plaso tools as a user would run
them. To run the end-to-end tests you'll need:

* the end-to-end test runner script, which is `tests/end-to-end.py`
* end-to-end tests configuration, for example `config/end-to-end.ini`
* location of the reference data, to compare test output against
* location of the result data
* location of the source data

For example, to run the predefined end-to-end tests:

```
PYTHONPATH=. python ./tests/end-to-end.py --debug -c config/end-to-end.ini --references-directory test_reference_data --results-directory test_results --sources-directory test_source_data
```

The predefined end-to-end tests are also run automatically on [GitHub](https://github.com/log2timeline/plaso/actions)
after every commit (code submit).

### Running end-to-end tests with Docker

To run end-to-end tests with Docker you'll need:

* Linux with Docker installed
* the end-to-end test Docker script, which is `config/end_to_end/run_tests_with_docker.sh`
* location of the end-to-end tests configurations
* location of the result data
* location of the source data

```
./config/end_to_end/run_tests_with_docker.sh /greendale/configs /greendale/sources /greendale/results
```
Log2timeline timeline generation: How to write a SQLite plugin

# How to write a SQLite plugin

To write a SQLite plugin it is best to use [l2t_scaffolder](https://github.com/log2timeline/l2tscaffolder).
The scaffolder will ask you questions and guide you through setting up the
necessary files needed for the plugin.

## Locate/create test data

* Before writing the plugin you'll need to have a SQLite database file to test
  the plugin with. Either generate one, or have one that does not contain
  personal data (since it will get checked into the project).
* Start by listing out all the SQL commands you'll need to issue against the
  database beforehand. Try them out manually, see if they work and produce the
  data you are looking for.

## Creating all the files.

* Start by [installing the l2t_scaffolder tool](https://l2tscaffolder.readthedocs.io/en/latest/sources/user/Installation.html)
* Have your git repo for Plaso correctly setup (personal fork, see
  [here](Developers-Guide.md)).
* Then follow the [usage instructions](https://l2tscaffolder.readthedocs.io/en/latest/sources/user/Using-The-Tool.html).
  * Essentially run this command (you'll need to remember the path to the test
    file and the path to the plaso git repo before you start).

```
$ l2t_scaffolder.py plaso
```

* After answering all questions a new feature branch will be created in your
  Plaso repository with all the files needed for the plugin.

## Write minimal tests

* Write a test that loads your plugin and parses a file.
* It will fail initially, but running the test while you're developing your
  plugin gives you a quick way to see if your code is doing what you expect.

## Develop plugin

* There will be TODO's and missing code inside the newly generated files. Fill
  these in with your code.

## Expand tests

* Add additional tests that test your plugin and formatter

## Register classes

* Edit `plaso/parsers/sqlite_plugins/__init__.py` to correct alphabetical
  order of the imports.
* Edit `plaso/formatters/__init__.py` to correct alphabetical order of imports.

## Extend the timeliner and formatters configurations

Extend the timeliner and formatters configurations with the necessary
definitions for new event data types.

See: [How to write a parser or (parser) plugin](How-to-write-a-parser.md)

## Code review/submit
Log2timeline timeline generation: How to write a tagging rule

# How to write a tagging rule

This page is intended to give you an introduction into writing rules for
tagging files that can be used by the [tagging analysis plugin](../user/Analysis-plugin-tagging.md).

Tagging rules use [event filters](../user/Event-filters.md).

The following script provides an overview of the attributes event data types
define.

```
PYTHONPATH=. ./utils/export_event_data.py
```

## Contributing changes to tagging rules

Once you have written or changed one or more tagging rules and want to
contribute th changes to the Plaso project. Please follow these steps:

1. If you have not already, please first read the developers guide on [contributing code](Developers-Guide.html#contributing-code).
2. Create a new feature branch.
3. If you are adding a new tagging rule, add it to the most appropriate tagging file. Make sure the tagging rule has brief comment about its purpose.
4. Make sure new or changed tagging rule have the necessary test coverage in https://github.com/log2timeline/plaso/tree/main/tests/data. This is to prevent tagging rules to silently stop working when the Plaso codebase change.
5. Add or update the [tagging rules](../user/Tagging-Rules.md) documentation.
Log2timeline timeline generation: Style guide

# Style Guide

Plaso follows the [log2timeline style guide](https://github.com/log2timeline/l2tdocs/blob/main/process/Style-guide.md).

## Plaso specific style points

### Event data attribute containers

#### Data types

Every event data attribute container defines a data type (DATA_TYPE).

Conventions for the data type names are:

1) If the data type is operating system (or operating system convension such as
POSIX) specific start with the name of operating system or convention.
Currently supported prefixes:

* android
* chromeos
* ios
* linux
* macos
* windows

Otherwise skip the operating system prefix.

2) Next is the name of the application, sub system or data format for example
'chrome', 'windows:registry' or 'windows:evtx'.

TODO: describe which one is preferred and why.

3) What follows are application, sub system or data format specific type
information for example 'windows:evtx:record'.

#### Value types

Values stored in an event data attribute container must be of certain types
otherwise event filtering or output formatting can break. Supported Python types
are:

* bool (also see note below)
* int
* str

A list, of the types previously mentioned types, are supported. **Do not use
dict or binary strings.**

Use a bool sparsely. For now it is preferred to preserve the original type.
For example if -1 represents False and 0 True, store the value as an integer
not as a bool. The message formatter can represent the numeric value as a
human readable string.

### Tests

* Use  the test functions available in the local test_lib.py as much as possible
 nstead of writing your own test functions. If you think a test function is
 missing please add it, or mail the developer list to see if you can get someone
 else to do it.
* Use `self.CheckTimestamp` for testing timestamp values.

Common test code should be stored in "test library" files, for example. the parser test
library is `tests/parsers/test_lib.py`.

We do this for a few reasons:

* to remove code duplication in "boiler plate" test code;
* to make the tests more uniform in both look-and-feel but also what is tested;
* improve test coverage;
* isolate core functionality from tests to prevent some future core changes
affecting the parsers and plugins too much.
Log2timeline timeline generation: How to write an analysis plugin

# How to write an analysis plugin

## Create file and class
* Plugin file in `plaso/analysis/`
  * Create an empty subclass of [AnalysisPlugin](../api/plaso.analysis.html#plaso.analysis.interface.AnalysisPlugin)
  * Register it with the analysis plugin by calling
   [AnalysisPluginManager.RegisterPlugin](../api/plaso.analysis.html#plaso.analysis.manager.AnalysisPluginManager.RegisterPlugin)
* Test file in `tests/analysis/`
  * Create an empty subclass of `tests.analysis.test_lib.AnalysisPluginTestCase`

## Write minimal tests
* Write a test that loads your plugin
* It will fail initially, but running the test while you're developing your
plugin gives you a quick way to see if your code is doing what you expect.

## Develop plugin
* Implement your subclass of [AnalysisPlugin](../api/plaso.analysis.html#plaso.analysis.interface.AnalysisPlugin)
* You'll need to define/override:
  * NAME
  * ExamineEvent()
  * CompileReport()
* You may also want to override:
  * URLS
  * ENABLE_IN_EXTRACTION, if your plugin is eligible to run while Plaso is
extracting events.

## Expand tests
* Add additional tests that test your plugin

## Register classes
* Edit `plaso/analysis/__init__.py` to import your plugin in the correct
 alphabetical order.

## Code review/submit
Log2timeline timeline generation: Packaging with docker

## Packaging with Docker

To create a Docker packaged release from the development release you also need:

* Docker

### Packaging

Build Plaso Docker image:

```
./utils/build_docker.sh
```

This will create a local Docker image: log2timeline/plaso:latest
Log2timeline timeline generation: How to write an output module

# How to write an output module

## Create file and class

* Plugin file in `plaso/output/`
  * Create an empty subclass of [plaso.output.interface.OutputModule](../api/plaso.output.html#plaso.output.interface.OutputModule)
  * Register it with the output module manager by calling
 [OutputManager.RegisterOutput](../api/plaso.output.html#plaso.output.manager.OutputManager.RegisterOutput)
* Test file in `tests/output/`
  * Create an empty subclass of `tests.output.test_lib.OutputModuleTestCase`

## Write minimal tests

* Write a test that loads your output module.
* It will fail initially, but running the test while you're developing your
plugin gives you a quick way to see if your code is doing what you expect.

## Develop plugin

* Implement your subclass of [plaso.output.interface.OutputModule](../api/plaso.output.html#plaso.output.interface.OutputModule)
* You'll need to define/overwrite:
  * NAME
  * DESCRIPTION
  * WriteEventBody
* You may also want to override:
  * Open()
  * Close()
  * GetMissingArguments()
  * WriteHeader()
  * WriteFooter()

## Expand tests

* Add additional tests that test your plugin

## Register classes

* Edit `plaso/output/__init__.py` to import your plugin in the correct
alphabetical order.

## Code review/submit

* Create a PR to have the changes reviewed and merged with the main branch.