deardaniel/secdata-raw huggingface dataset:
Log2timeline timeline generation: Developing Windows

# Developing on Windows

## Git

To download the latest version of Plaso you'll need to install the
[git tools](https://git-scm.com/downloads).

Checkout the Plaso source from the git repo:

```bash
git clone https://github.com/log2timeline/plaso.git
```

**If you intend to submit code make sure to configure git to use convert to the
Unix-style end-of-line characters (linefeed) on submission and not have the
Windows-style end-of-line characters (carriage return + linefeed).**

We recommend to configure your editor of choice to use linefeed only and turn
off git's autocrlf:

```bash
git config --global core.autocrlf false
```

## Dependencies

Install the dependencies, using the
[prepackaged dependencies](Development-Dependencies.html#Windows).

Check if you have all the dependencies installed and have the right minimum
version:

```bash
C:\Python3\python.exe utils\check_dependencies.py
```

**Note that some dependencies are actively under development and can be
frequently updated, therefore we recommend checking the status of the
dependencies regularly.**

## Running the development code

To run the development code directly from source make sure Python can find the
Plaso source files by setting PYTHONPATH correspondingly.

```bash
set PYTHONPATH=C:\plaso-build\plaso
```

To run a tool, for example `pinfo`:

```bash
C:\Python3\python.exe C:\plaso-build\plaso\plaso\frontend\pinfo.py timeline.plaso
```

## Development tools

If you intend to do development on Plaso you'll also need to install the
following development tools:

* pylint
Log2timeline timeline generation: Developing Virtualenv

# Developing in a Virtualenv

For development purposes, Plaso can be installed using virtualenv.

**Note that this is intended for development use only, and if you aren't
comfortable debugging package installation, this is not for you.**

## Fedora

### Install virtualenv

To install virtualenv on Fedora (or equivalent) run:
```
sudo dnf install python-virtualenv
```

### Installing build dependencies

**TODO add more text**

## Ubuntu

### Installing virtualenv

To install virtualenv on Ubuntu (or equivalent) run:
```
sudo apt-get install python-virtualenv
```

### Installing build dependencies

**TODO add more text**
```
sudo apt-get install libyaml-dev liblzma-dev
```

## MacOS

### Installing virtualenv

If you have `pip` setup on your system, you can install
`virtualenv` using:

```bash
pip install virtualenv
```

To install `pip`, either install a version of Python 3.6+ or,
using [homebrew](https://brew.sh/), run:

```bash
brew install python@3
```

### Installing build dependencies

**TODO**

The `pycrypto` dependency requires the `gmp` library. This is
installable with homebrew:

```bash
brew install gmp
```

Once setup, we need to add the paths for the system to locate
the `gmp` library. This is accomplished during the
`pip install -r requirements.txt` stage of the next section.
Before running the `pip install ...` command, please run:

```bash
export "CFLAGS=-I/usr/local/include -L/usr/local/lib"
```


## Setting up Plaso in a virtualenv

1. Create a virtualenv called 'plasoenv' `virtualenv plasoenv`
1. Activate the virtualenv: `source ./plasoenv/bin/activate`
1. Update pip (Note that using pip outside virtualenv is not recommended as it
ignores your systems package manager.): `pip install --upgrade pip`
1. Install the Python requirements
    ```bash
    # Where 'plaso' is your local Plaso source directory
    cd plaso
    curl -O https://raw.githubusercontent.com/log2timeline/plaso/main/requirements.txt
    pip install -r requirements.txt
    ```
1. Once you finish you development session, deactivate virtualenv: `deactivate`
Log2timeline timeline generation: Developing Fedora

# Developing on Fedora

To download the latest version of Plaso you'll need to install the git tools:
```bash
sudo dnf install git
```

Checkout the plaso source from the git repo:
```bash
git clone https://github.com/log2timeline/plaso.git
```

## Dependencies

Install the dependencies, using either:
* A [virtualenv](Developing-Virtualenv.html#Fedora)
* The [prepackaged dependencies](Development-Dependencies.html#Fedora).

Check if you have all the dependencies installed and have the right minimum
versions:
```
python utils/check_dependencies.py
```

**Note that some dependencies are actively under development and can be
frequently updated, therefore we recommend checking the status of the
dependencies regularly.**

## Development tools
If you intend to do development on Plaso you'll also need to install some
development tools:

* PyLint 3.2.x
Log2timeline timeline generation: Profiling

Plaso supports various profiling options for troubleshooting and performance
tuning.

## Profiling CPU usage

The CPU usage of various parts of Plaso its procseeing can be profiled with the
CPU ussage profiler.

To profile the CPU usage run log2timeline.py with the following options:

```bash
log2timeline.py --profilers=${PROFILERS} --profiling-directory=profile --storage-file timeline.plaso image.raw
```

Where ${PROFILERS} is comma separated list of one or more of the following
CPU usage profilers:

Name | Description
--- | --- 
analyzers | Profile CPU time of analyzers, like hashing
parsers | Profile CPU time per parser
processing | Profile CPU time of processing phases
serializers | Profile CPU time of serialization

## Profiling memory usage

The memory usage of the main (foreman) and worker processes can be profiled with
the memory profiler.

To profile the memory usage run log2timeline.py with the following options:

```bash
log2timeline.py --profilers=memory --profiling-directory=profile --storage-file timeline.plaso image.raw
```

## Profiling storage

The amount of data read and / or written by the storage con be profiled with
the storage profiler.

To profile the storage run log2timeline.py with the following options:

```bash
log2timeline.py --profilers=storage --profiling-directory=profile --storage-file timeline.plaso image.raw
```

## Profiling the task queue

The task queue profiler tracks:

* number of tasks queued for processing or to be merged
* number of tasks processing
* number of tasks pending to be merged
* number of tasks abandoned
* total number of tasks, included completed tasks

To profile the task queue statue run log2timeline.py with the following options:

```bash
log2timeline.py --profilers=task_queue --profiling-directory=profile --storage-file timeline.plaso image.raw
```

## Graphing profiles

To graph profiling data you will need to have the matplotlib and numpy Python
modules installed.

### Graphing CPU usage over time

`./utils/plot-cpu-usage.py profile`

### Graphing memory usage over time

`./utils/plot-memory-usage.py profile`

### Graphing task queue over time

`./utils/plot-task-queue.py profile`

### Also see

* [Troubleshooting Plaso Issues - Memory Edition](http://blog.kiddaland.net/2014/11/troubleshooting-plaso-issues-memory.html)
Log2timeline timeline generation: tag macos

# Tags MacOS application execution events.
application_execution
  data_type is 'macos:application_usage:entry'
  data_type is 'syslog:line' AND body contains 'COMMAND=/bin/launchctl'

# Tags MacOS application installation events.
application_install
  data_type is 'plist:key' AND plugin is 'plist_install_history'

# Tags creation events of HFS/HFS+ files with a filename that contains LaunchAgents as a path segment and plist as extension.
autorun
  data_type is 'fs:stat' AND timestamp_desc is 'Creation Time' AND filename contains PATH('LaunchAgents') AND filename contains '.plist'

# Tags downloaded file events.
file_download
  data_type is 'chrome:history:file_downloaded'
  data_type is 'macos:lsquarantine:entry'
  timestamp_desc is 'Downloaded Time'

# Tags device connection events.
device_connection
  data_type is 'ipod:device:entry'
  data_type is 'plist:key' AND plugin is 'plist_airport'

# Tags OLE compound document print events.
document_print
  data_type is 'olecf:summary_info' AND timestamp_desc is 'Last Printed Time'
Velociraptor DFIR platform: security

```
Contact: security@rapid7.com
Encryption: https://keys.openpgp.org/search?q=396F6DCA9B60EB1AF2D1621FA885DF1431A0A489
Preferred-Languages: en
Canonical: https://www.rapid7.com/.well-known/security.txt
Policy: https://www.rapid7.com/security/disclosure/
```
Velociraptor DFIR platform: README

# Velociraptor docker container

This directory builds a docker container for launching Velociraptor.

This container is designed for a couple of use cases:

1. No prior Velociraptor deployment. Spin up Velociraptor easily with
   default everything.

   This use case creates a new configuration file with:
   * Self signed certificates
   * GUI port by default is listening on 8889, Frontend port listening on 8000
   * Generate a new configuration file stored in the /etc/ directory
   * Mounts the datastore in the /datastore/ - this allows the deployment data to persist.
   * The deployment will trigger a build for client assets like MSI, Deb and RPM packages
   * An initial user is created with admin permissions. Default
     password is `password` or take from the .env file, but you should
     change it from the GUI after the server is up.

2. An existing Velociraptor deployment with a pre-configured configuration file.

   In this case, simply copy your configuration file to the `etc/`
   directory and adjust the .env file to match the forwarded ports.


In both cases the `datastore` directory is used as permanent storage
and remains after the container is terminated. It is safe to delete
the datastore and start fresh at any time - clients will just
re-connect and re-enrol.


## Quick start

The CI pipeline uploads the container to the GitHub container
registry, so all you need to do is copy the `compose.yaml` from this
directory and simply run:

```
docker-compose up
```

The latest image will be fetched from the registry, a default
configuration will be generated and the server will be started.

You can connect to the GUI on `https://localhost:8889/` with default
password of `password`. You can tweak the `.env` file to update this
default password.

The datastore files will be stored in the `datastore` directory and
the generated config file will be stored in `etc`. Make sure to back
up the generated config file to ensure existing clients can still talk
to this server.
Velociraptor DFIR platform: Connection incoming


248257009	Machine1	05-08-2020 13:36:21	05-08-2020 13:38:33	yolo	RemoteControl	{e03e5706-1972-4ced-b209-3fa9b230d62b}	
248257009	Machine1	05-08-2020 13:39:53	05-08-2020 13:42:44	yolo	RemoteControl	{af03e0b1-85c3-49b2-8074-c66ce0fd5f4a}
Velociraptor DFIR platform: README

## Velociraptor Golden Tests

The files in this directory are the golden test suite used by the CI
pipeline.

What are Golden tests? Golden testing is a methodology to quickly and
efficiently write tests:

1. First a test case is written with the VQL queries that should be
   run. These queries are written in a file with a `.in.yaml`
   extension.
2. The `golden` test runner can be run on the test files using `make
   golden` at the top level of this repository.
3. If the output of the queries is different from the existing output
   (stored in `.out.yaml` files) the test will fail. The golden runner
   will then update the output file with the new data.
4. The user can compare the changes in the output file (e.g. using
   `git diff`) and if the changes are OK then simply `git add` the new
   output file. Running the golden tests again should produce no
   change.

By default the makefile rule runs the debug race detector binary (you
can built this using just `make` at the top level. This will produce a
debug build in `./output/velociraptor`. This binary includes the race
detector and so it is quite slow to run but worth it for tests.

If you find you need to iterate quicker you can manually run the
production binary (built using `make linux`) by modifying the command
run by the `make golden` command.

Additionally you can run the `dlv` debugger in the golden output by
running `make debug_golden` at the top level.

To filter the test cases (so they don't have to all run) you can set
the `GOLDEN` environment variable. For example to only run the tests
in `pe.in.yaml`:

```
$ GOLDEN=pe make golden
./output/velociraptor -v --config artifacts/testdata/windows/test.config.yaml golden artifacts/testdata/server/testcases/ --env srcDir=`pwd` --filter=pe
```


## NOTES

Golden Testing requires the output to not change between subsequent
runs and when running between different environment. This means that
output that naturally changes should be avoided - for example output
that depends on:

- Time
- File paths
- Operating systems

You can use a combination of mocking plugin output and selecting
specific columns to format the output in such a way that it does not
depends on ephemeral things.


## Developing artifacts

When developing artifacts using TDD it is useful to load the raw
artifact YAML without needing to build the binary each time. This way
we can iterate over the artifact yaml and see the results immediately
in the golden out yaml.

An example command line is:

```
./output/velociraptor-v0.7.0-linux-amd64 -v --config artifacts/testdata/windows/test.config.yaml golden artifacts/testdata/server/testcases/ --env srcDir=`pwd` --filter=hostsfile --definitions artifacts/definitions/Generic/System/
```

Here the binary will force load the raw yaml definition at runtime
overriding the built in artifact definition. It will then run the
Golden test `hostsfile.in.yaml`
Velociraptor DFIR platform: gopsutil LICENSE

gopsutil is distributed under BSD license reproduced below.

Copyright (c) 2014, WAKAYAMA Shirou
All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

 * Redistributions of source code must retain the above copyright notice, this
   list of conditions and the following disclaimer.
 * Redistributions in binary form must reproduce the above copyright notice,
   this list of conditions and the following disclaimer in the documentation
   and/or other materials provided with the distribution.
 * Neither the name of the gopsutil authors nor the names of its contributors
   may be used to endorse or promote products derived from this software without
   specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


-------
internal/common/binary.go in the gopsutil is copied and modified from golang/encoding/binary.go.



Copyright (c) 2009 The Go Authors. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

   * Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
   * Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
   * Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Velociraptor DFIR platform: sample

instance of __InstanceCreationEvent
{
        TargetInstance =
instance of Win32_Process
{
        Caption = "notepad.exe";
        CommandLine = "notepad";
        CreationClassName = "Win32_Process";
        CreationDate = "[PII_REDACTED]1847.788310-420";
        CSCreationClassName = "Win32_ComputerSystem";
        CSName = "DESKTOP-IOME2K5";
        Description = "notepad.exe";
        ExecutablePath = "C:\\Windows\\system32\\notepad.exe";
        Handle = "984";
        HandleCount = 236;
        KernelModeTime = "468750";
        MaximumWorkingSetSize = 1380;
        MinimumWorkingSetSize = 200;
        Name = "notepad.exe";
        OSCreationClassName = "Win32_OperatingSystem";
        OSName = "Microsoft Windows 10 Pro N|C:\\WINDOWS|\\Device\\Harddisk0\\Partition2";
        OtherOperationCount = "151";
        OtherTransferCount = "2558";
        PageFaults = 3939;
        PageFileUsage = 2968;
        ParentProcessId = 2424;
        PeakPageFileUsage = 2968;
        PeakVirtualSize = "[PII_REDACTED]816";
        PeakWorkingSetSize = 15196;
        Priority = 8;
        PrivatePageCount = "3039232";
        ProcessId = 984;
        QuotaNonPagedPoolUsage = 14;
        QuotaPagedPoolUsage = 256;
        QuotaPeakNonPagedPoolUsage = 15;
        QuotaPeakPagedPoolUsage = 257;
        ReadOperationCount = "2";
        ReadTransferCount = "9672";
        SessionId = 1;
        ThreadCount = 6;
        UserModeTime = "156250";
        VirtualSize = "[PII_REDACTED]720";
        WindowsVersion = "10.0.17134";
        WorkingSetSize = "15560704";
        WriteOperationCount = "0";
        WriteTransferCount = "0";
        Foo = {1, 2, 3, 4};
        AcceptPause = FALSE;
        AcceptStop = TRUE;
        NullDescription = NULL;
};
        TIME_CREATED = "[PII_REDACTED]7753198";
};
Velociraptor DFIR platform: style

# Velociraptor code base style guide

The following summarises at a high level some style conventions in the
codebase.

## Receivers shall be named `self`

This convention is contrary to the usual Golang convention of using
different variable names for the receiver. In Velociraptor, all
receivers are named `self` in a similar way to the Python convention.

Using the same consistent name for all receivers makes code much
easier to read and reduces cognitive load. When you see `self` you
know it refers to the receiver of the current method.

## Variable naming

1. Local variables are named in snake case format. e.g: `fs_type`
2. Public Methods, functions and Types are named using Upper Camel
   Case. e.g. `MatchComponentPattern()`, `FSPathSpec`
3. Private Methods , Functions and types are names using lower camelCase

4. Struct methods should have the receiver defined as a pointer.

   This avoids copying the receiver on entering the function. The
   default golang behavior is responsible to many bugs (for example
   copying mutexes leading to race conditions). Additionally copying
   receivers may actually hide performance issues.

   It is just safer to pass receivers as pointers.

## Locking semantics

1. Minimize the use of recursive locks, preferring to use `sync.Mutex`
   instead.

2. Mutexes shall be embedded in the structs they protect as a private
   variable. The variable shall be named `mu`. Hence locks are
   obtained using `self.mu.Lock()`

3. Public methods on structs that require locking shall obtain the
   lock themselves. If the struct needs to call internal private
   methods, by convention these should not acquire the lock.

   This implies that generally a method that starts with a lower case
   can only be called with the locks held (since they are private
   methods).

   Methods starting with a capital letter are safe to call without
   holding the lock.

For example consider the following struct:

```go
type ExampleType struct {
  mu           sync.Mutex
  ...
}

func (self *ExampleType) PublicMethod() {
  self.mu.Lock()
  defer self.mu.Unlock()

  self.privateMethod()
}

func (self *ExampleType) privateMethod() {
  .. do things with the lock acquired ...
}

func (self *ExampleType) otherMethod() {
  // Safe to call the private method which assume lock is held.
  self.privateMethod()
}
```

This scheme makes it easier to see which methods require the locks
held and easier to quickly audit the methods called within a function
to ensure they are safe:

* If a method starts with a lower case (private method) we know the
  lock must be held on entry to the method.
* All methods called from this method must also start with lower case.

Consistency in naming here ensures we do not deadlock due to recursive
locks.

## General guidelines

* Never use `panic()` in production code
Velociraptor DFIR platform: README

# Building the resources

The resources are built once using go-winres
https://github.com/tc-hib/go-winres

The syso files are checked into the repo and reused on each build.

```
cd ./docs/
go-winres make
mv rsrc_windows_386.syso rsrc_windows_amd64.syso ../bin/
```
Velociraptor DFIR platform: README

# Setting up monitoring.

To set up monitoring for Velociraptor servers you will need to install
both Graphana and Prometheus.

## Setting up Prometheus

Prometheus will scrape the Velociraptor server monitoring port and
record time series of critical server state. You can use the provided
prometheus.yaml file without change and launch prometheus like this:

```
$ prometheus  --config.file velociraptor.yml
```

## Setting up Graphana

Graphana is a graphing package which helps to visualize the Prometheus
data. Graphana is very easy to use and can be configured using its web
interface.

First add Prometheus as a data source.

You can import the graphana.json file as a new dashboard. This is a
good starting point for a useful monitoring dashboard and will show
the following graphs:

1. Flow completion rate in flows/second - this gives an indication of
   active hunts and how busy the server is.

2. Client current connections - shows how many clients are currently
   connected to the server.

3. Resident memory size - shows how much memory the server is using.

4. CPU Load - shows how much cpu the server is currently using.