Saravanan666/pypi-org huggingface dataset:
Package: karton-archive-extractor
Extractor karton servicePerforms extraction of known archive types and e-mail attachments. Produces "raw" artifacts for further classification.Author: CERT.plMaintainers: psrok1, nazywamConsumes:{
    "type":  "sample",
    "stage": "recognized",
    "kind":  "archive"
    "payload": {
        "sample": <Resource>,
        "extraction_level": <int, default: 0>,
        "password": <archive password>,
    }
}Produces:{
    "type": "sample",
    "kind": "raw",
    "payload": {
        "sample": <Resource>,
        "parent": <Resource>,
        "extraction_level": <int++>
    }
}UsageFirst of all, make sure you have setup the core system:https://github.com/CERT-Polska/kartonIn order to unpack all available formats you'll also need a few native dependencies that sflock relies on, the installation method recommended by sflock is:RUNsed-i's/ main/ main non-free/'/etc/apt/sources.list\&&apt-getupdate&&apt-getinstall-y\p7zip-full\rar\unace\cabextract\lzipThen install karton-archive-extractor from PyPi:$pipinstallkarton-archive-extractor

$karton-archive-extractorConfigurationThere are several configuration options you can tweak up to your liking.[archive-extractor]# Maximum levels of nested extractionmax_depth=5# Maximum unpacked child filesize, larger files are not reportedmax_size=26214400# Maximum number of children files for further analysismax_children=1000To learn more about configuring your karton services, take a look atkarton configuration docsRunning in DockerSflock usesZipJailas a usermode syscall filtering mechanism. As a result, in our experience, container running the karton service has to have theSYS_PTRACEcapability in order for the ptrace to execute correctly. Make sure it's enabled if you run into problems extracting certain archive types.Supported archive/compression formats*.7z
.ace
.bup
.cab
.daa
.eml
.gz
.gzip
.iso
.lha
.lz
.lzh
.msg
.mso
.pdf
.rar
.tar
.tar.bz2
.tar.gz
.udf
.vhd
.vhdx
.xz
.zip* Assuming you are running Linux, please see thesflock's readmefor more informationPE files debloatingSome malicious PE files contain intentionally added junk to make them too big for processing. Starting from v1.4.0, archive extractor
supports optional debloating of these files, usingdebloat tool made by Squiblydoo.certpl/karton-archive-extractorDocker image debloats PE files by default. To enable debloating in
karton-archive-extractor installed from PyPI, you need to install additional extra dependencies:pip install karton-archive-extractor[debloat]
Package: karton-asciimagic
AsciiMagic karton serviceExtracts next stages of various ASCII files that contain malware, for example hex, base64, etc.Author: CERT.plMaintainers: ola, msm, nazywamConsumes:{"type":"sample","stage":"recognized","kind":"ascii"}Produces:{"type":"sample","kind":"runnable","stage":"recognized","platform":"win32","extension":"exe",},{"type":"sample","kind":"raw"}UsageFirst of all, make sure you have setup the core system:https://github.com/CERT-Polska/kartonThen install karton-asciimagic from PyPi:$pipinstallkarton-asciimagic

$karton-asciimagic
Package: karton-autoit-ripper
AutoIt-Ripper karton serviceUsesAutoIt-Ripperto automatically extract embedded AutoIt scripts and resources from compiled binaries.Author: CERT.plMaintainers: nazywamConsumes:{"type":"sample","stage":"recognized","kind":"runnable","platform":"win32"},{"type":"sample","stage":"recognized","kind":"runnable","platform":"win64"}Produces:{"type":"sample","kind":"raw"},{"type":"sample","kind":"script","stage":"analyzed","extension":"au3",}UsageFirst of all, make sure you have setup the core system:https://github.com/CERT-Polska/kartonThen install karton-autoit-ripper from PyPi:$pipinstallkarton-autoit-ripper

$karton-autoit-ripper
Package: karton-classifier
Classifier karton serviceFile type classifier for the Karton framework.Entrypoint for samples. Classifies type of samples labeled askind: raw,
which makes them available for subsystems that receive samples with specific
type only (e.g.raw=>runnable:win32:exe)Author: CERT.plMaintainers: psrok1, msm, nazywamConsumes:{
    "type": "sample",
    "kind": "raw"
    "payload": {
        "magic":  "output from 'file' command",
        "sample": <Resource>
    }
}Produces:{
    "type":      "sample",
    "stage":     "recognized",
    "kind":      "runnable"  # Executable format default for OS platform
              || "document"  # Office document
              || "archive"   # Archive containing samples (zip, e-mails)
              || "dump"      # Dump from sandbox
              || "script",   # Script (js/vbs/bat...)
              || "misc",     # No platform or extension
    "platform":  "win32" 
              || "win64" 
              || "linux" 
              || "android",
              || "macos",
              || "freebsd",
              || "netbsd",
              || "openbsd",
              || "solaris",
    "extension": "*",        # Expected file extension
    "mime": "*",        # Expected file mimetype
    ... (other fields are derived from incoming task)
}Warningthe outputmimefield is not deterministic across libmagic versions and can change depending on the version you're using. We don't recommend creating consumers that listen on it directly.UsageFirst of all, make sure you have setup the core system:https://github.com/CERT-Polska/kartonThen install karton-classifier from PyPi:$pipinstallkarton-classifier

$karton-classifier
Package: karton-config-extractor
Config-extractor karton serviceExtracts static configuration from samples and memory dumps using the malduck engine.Author: CERT.plMaintainers: nazywam, psrok1, msmConsumes:{
    "type": "sample",
    "stage": "recognized",
    "kind": "runnable",
    "platform": "win32"
},
{
    "type": "sample",
    "stage": "recognized",
    "kind": "runnable",
    "platform": "win64"
},
{
    "type": "sample",
    "stage": "recognized",
    "kind": "runnable",
    "platform": "linux"
},
{
    "type": "analysis",
},Whilesampletype is self explanatory, theanalysistype might be confusing. Theanalysistask is an output from
one of sandboxes:drakvuf-sandbox,cuckoo, orjoesandbox. Analysis is asamplewith additional memory dumps
attached.Theanalysistype task is expected to be in format:task = Task(
    headers={"type": "analysis"}
    payload={
        "sample": <sample>,
        "dumps.zip": Resource.from_directory("dumps.zip", dumps_path.as_posix()),
        "dumps_metadata": [
            {"filename": <dump1_filename>, "base_address": <dump1_base_address>},
            {"filename": <dump2_filename>, "base_address": <dump2_base_address>},
            {"filename": <dump3_filename>, "base_address": <dump3_base_address>},
            [...]
        ],
    }
)wheredumps_metadatacontains information about filename and base address for every memory dump indumps.zip. The
following attributes are:filenamewhich is relative path to the dumps.zip contents;base_addresswhich hex-encoded base address for dump (leading0xis supported)
You can specify multiple entries for the same file if the same memory dump was found on different base addresses.The extractor tries to retrieve config from each memory dump and will pick only the best candidate from each malware
family.Produces:# Dropped dumps related with static configuration
{
    "type": "sample",
    "stage": "analyzed",
    "kind": "dump",
    "platform": "win32",
    "extension": "exe"
    "payload": {
        "sample": <Resource>, # Dump where config was found
        "parent": <Resource>  # Original executable
    }
}

# Static configuration
{
    "type": "config",
    "family": <str>, # Family name
    "payload": {
        "config": <dict>,     # Static configuration
        "sample": <Resource>, # Dump where config was found
        "parent": <Resource>, # Original executable
    }
}UsageFirst of all, make sure you have setup the core system:https://github.com/CERT-Polska/kartonThen install karton-config-extractor from PyPi:$pipinstallkarton-config-extractor

$karton-config-extractor--modulesmalduck-extractor-modules/
Package: karton-dashboard
Karton DashboardA small Flask application for Karton task and queue introspection.UsageFirst of all, make sure you have setup the core system:https://github.com/CERT-Polska/kartonThen install karton-dashboard from PyPi:$pipinstallkarton-dashboard

$karton-dashboardrun-h127.0.0.1-p5000Thekarton-dashboardis just a wrapper on theflaskprogram, and it works with any arguments accepted by flask. For examplekarton-dashboard --help, orkarton-dashboard run -h 0.0.0.0 -p 1234. Seeflask documentationfor more information.Xref buttonsIf you have other systems that store artifacts related to a specific karton task there's an option to easily link them in the task view.You'll need to set thexrefsoptions ofdashboardsection in yourkarton.iniconfiguration file like so:[dashboard]xrefs={"mwdb": "https://mwdb.cert.pl/?q=karton:{root_uid}"}Each link will be rendered as a button in the task view like so:MetricsKarton tracks number of consumed, produced and crashed tasks for each service (identity).Karton-dashboard exposes this information (in addition to some other task/queue statistics) on the/varzendpoint using Prometheus data format.You can use this data to easily build custom karton dashboards and setup alerts for unusual events or high loads. Just point the data source URL tohttps://karton-dashboard/varzand configure the dashboard however you like.Here are a few examples of how the data can be digested and presented in a grafana dashboard:sum(karton_tasks{name!=""}) by (priority)sum(karton_tasks{name!=""}) by (status)sum(karton_tasks{name!=""}) by (name)
Package: karton-memory-watcher
karton memory watcherWorking for several months in a row, tasks gradually become more and more significant in terms of RAM consumption.
And sometimes forcing a call to the garbage collector doesn't help.
This is especially felt when working with libraries that help to parse certain file formats -
at the start, the process takes up 60 megabytes of RAM, and after a couple of months - 400.This library is a watcher that monitors the gradual increase in the service's RAM usage and shuts
it upon reaching a set threshold.For what ways to run karton consumer is it suitable?For every way that uses autorestart. For example, I can suggest at least three of them:systemctl service restart policy:on-failure/alwaysdocker service restart policy:on-failure/always/unless-stoppedScreen infinite loop:while true; do your_app; sleep 10; doneInstallationYou can install it via pip:python3 -m pip install karton-memory-watcherHow to use itSimple way:fromkarton.coreimportConsumerfromkarton.memory_watcherimportimplant_watcher,RestartBehavior,RestartRule...classYourConsumer(Consumer):passif__name__=="__main__":foobar=YourConsumer()implant_watcher(foobar,RestartRule(extra_consumed_memory_percent=80,# call_before_exit=(close_db_connections, )))foobar.loop()RestartRule modes:proceed_tasks: count of tasks to proceed for restartelapsed_time: how many seconds should passextra_consumed_memory_percent: extra memory used in % (100% means twice of memory compared to point before starting first task). It uses megabytes to calculate percents.extra_consumed_megabytes: extra memory used in megabytes (e.g. your service at start uses 60MB and you need to kill it if it consumes extra 500+ MB)You can see usage cases intests.Alternatives?Systemctl:In January 2023 there is a nice feature request about MemoryMax policy:https://github.com/systemd/systemd/issues/25966Also writing your own agent that will track memory leaks and send systemctl service to restart (but be careful not to kill ongoing tasks!).Dockerwip, feel free to suggest how to gently restart container to allow task normally proceed
Package: karton-misp-pusher
karton-misp-pusherListens for new samples in thekartonpipeline and uploads them to MISP.Configs are parsed using themwdb-iocextractproject. This means, that we operate on a higher level than raw JSON configs, and makes
it possible to correlate different samples and campaigns (for example, by the used crypto
material).Author: CERT.plMaintainers: msmConsumes:{
    "type": "config",
}Result:UsageFirst of all, make sure you have setup the core system:https://github.com/CERT-Polska/karton.More info here.Then install karton-misp-pusher from PyPi:$pipinstallkarton-misp-pusher

$karton-misp-pusher--misp-urlhttps://misp.url--misp-keySECRET123You can also add optional xrefs to mwdb with--mwdb-url, or skip MISP
verification with--misp-insecure. For more options see--help.
Package: karton-mwdb-reporter
Reporter karton serviceUploads samples and static configs to malwaredbAuthor: CERT.plMaintainers: psrok1, nazywamConsumes:{
    "type": "sample",
    "stage": "recognized" || "analyzed" || "unrecognized"
},
{
    "type": "config"
},
{
    "type": "blob"
}Produces:(nothing)UsageFirst of all, make sure you have setup the core system:https://github.com/CERT-Polska/kartonModify yourkarton.iniconfig to include information about your mwdb-core instance and reporter credentials:[mwdb]api_url=http://mwdb.my-awesome-org/api/api_key=eyJhYWF....Instead of providingapi_keyyou can also useusername/passwordbut password-authenticated sessions are short-lived and service will need to re-auth from time to time.Then install karton-mwdb-reporter from PyPi:$pipinstallkarton-mwdb-reporter

$karton-mwdb-reporterConfigurationUsing the--report-unrecognizedflag you specify whether the reporter should upload files unrecognized by the classifier. You can also configure this using the built-in configuration backend by either adjusting it in the karton.ini[mwdb-reporter]report_unrecognized=trueor setting the environmental variable like soKARTON_MWDB-REPORTER_REPORT_UNRECOGNIZED=true.To learn more about configuring your karton services, take a look atkarton configuration docs
Package: karton-unpacker
Unpacker Karton ServiceA modularKarton Frameworkservice that unpacks common packers likeUPXand others using theQiling Framework.This project is FREE as in FREE :beer:, use it commercially, privately or however you see fit.If you like this project and wish to donate :moneybag: to support the fight against malware...Buy me a :tea:, as I don't drink :beer:, by sending me some ₿ to16oXesi7uv3jdPZxxwarHSD2f3cNMpaih9Figure 1: Example of UPX Unpacked ChildrenFigure 2:Qiling FrameworkUnpackingcalc.exeshellcode fromtests/shellcode.exeConsumes:{"type":"sample","stage":"recognized","kind":"runnable","platform":"win32"},{"type":"sample","stage":"recognized","kind":"runnable","platform":"win64"},{"type":"sample","stage":"recognized","kind":"runnable","platform":"linux"}{"type":"sample","kind":"runnable","stage":"recognized","platform":<win32|win64|linux>(IfPEFileinDump),"payload":{"sample":<Resource>,"parent":<Resource>,}}UsageMake sure you have setup the core system:https://github.com/CERT-Polska/kartonInstall from PyPi:$sudoaptinstall-ypython3-virtualenvpython-is-python3
$virtualenvvenv/
$sourcevenv/bin/activate
$pipinstallkarton-unpacker
$gitclonehttps://github.com/c3rb3ru5d3d53c/karton-unpacker-modules.gitmodules/
$findmodules/-name"requirements.txt"|whilereadi;dopipinstall-r$i;done$gitclone--recursivehttps://github.com/qilingframework/qiling.git# Due to distribution restriction, Qiling Framework will not bundle Microsoft Windows DLL files and registry.# Please use the script qiling/examples/scripts/dllscollector.bat on your Windows machine to collect the required DLLS for the rootfs# Once the required DLLs have been collected copy them in the rootfs$karton-unpacker--config-filekarton.ini--modulesmodules/--rootfsqiling/examples/rootfs/--timeout30--debugInstall from Source:$sudoaptinstall-ypython3-virtualenvpython-is-python3
$gitclone--recursivehttps://github.com/c3rb3ru5d3d53c/karton-unpacker.git
$cdkarton-unpacker/
$virtualenvvenv/
$sourcevenv/bin/activate
$pipinstall.
$gitclone--recursivehttps://github.com/qilingframework/qiling.git# Due to distribution restriction, Qiling Framework will not bundle Microsoft Windows DLL files and registry.# Please use the script qiling/examples/scripts/dllscollector.bat on your Windows machine to collect the required DLLS for the rootfs# Once the required DLLs have been collected copy them in the rootfs$karton-unpacker--config-filekarton.ini--modulesmodules/--rootfsqiling/examples/rootfs/--timeout30--debugTesting Your InstallationOnce you have completed installingkarton-unpacker, try uploading the filetests/shellcode.exeto mwdb.If successful, you will see a file in relations with the nameunpacked, this is the extracted shellcode to spawncmd.exe.ContributingIf you wish to contribute your own modules to automatically unpack malware, please refer toCONTRIBUTING.md
Package: karton-yaramatcher
YaraMatcher karton serviceScans analyses and samples with yara rules and spawns tasks with appropiate tags.Author: CERT.plMaintainers: nazywamConsumes:{"type":"sample","stage":"recognized","kind":"runnable"},{"type":"sample","stage":"recognized","kind":"dump"},{"type":"analysis","kind":"cuckoo1"},{"type":"analysis","kind":"drakrun"},{"type":"analysis","kind":"joesandbox"}Produces:{"type":"sample","stage":"analyzed"}UsageFirst of all, make sure you have setup the core system:https://github.com/CERT-Polska/kartonThen install karton-yaramatcher from PyPi:$pipinstallkarton-yaramatcherAnd run the karton service by pointing it to yourYARArules repository:$karton-yaramatcher--rulesyara_rule_directory
Package: kartothek
KartothekKartothek is a Python library to manage (create, read, update, delete) large
amounts of tabular data in a blob store. It stores data as datasets, which
it presents as pandas DataFrames to the user. Datasets are a collection of
files with the same schema that reside in a blob store. Kartothek uses a metadata
definition to handle these datasets efficiently. For distributed access and
manipulation of datasets Kartothek offers aDaskinterface.Storing data distributed over multiple files in a blob store (S3, ABS, GCS,
etc.) allows for a fast, cost-efficient and highly scalable data infrastructure.
A downside of storing data solely in an object store is that the storages
themselves give little to no guarantees beyond the consistency of a single file.
In particular, they cannot guarantee the consistency of your dataset. If we
demand a consistent state of our dataset at all times, we need to track the
state of the dataset. Kartothek frees us from having to do this manually.Thekartothek.iomodule provides building blocks to create and modify these
datasets in data pipelines. Kartothek handles I/O, tracks dataset partitions
and selects subsets of data transparently.InstallationInstallers for the latest released version are availabe at thePython
package indexand on conda.# Install with pippipinstallkartothek# Install with condacondainstall-cconda-forgekartothekWhat is a (real) Kartothek?A Kartothek (or more modern: Zettelkasten/Katalogkasten) is a tool to organize
(high-level) information extracted from a source of information.
Package: karvdash-client
Karvdash API Client and CLIKarvdash(Kubernetes CARV dashboard) is a dashboard service for facilitating data science onKubernetes. It supplies the landing page for users, allowing them to launch notebooks and other services, design workflows, and specify parameters related to execution through a user-friendly interface. Karvdash manages users, wires up relevant storage to the appropriate paths inside running containers, securely provisions multiple services under one externally-accessible HTTPS endpoint, while keeping them isolated in per-user namespaces at the Kubernetes level, and provides an identity service for OAuth 2.0/OIDC-compatible applications.Check out theuser guide and API documentation.The API client library provides a class to interact with a Karvdash installation using function calls. It also includes a command line tool,karvdash-client, to run respective API commands from the CLI.To use the client library you should provide the API endpoint and an authentication token in a configuration file (an example is shown in thePython library documentation). For services running within Karvdash-managed namespaces, this file is automatically created and mounted in pods at/var/lib/karvdash/config.ini.
Package: karyoplot
karyoplotWhat is karyoplot ?karyoplotis a python package developed to draw simple karyoplot/ideogram with a simple configuration file like Circos [1], without any lines of code like karyoploteR [2].karyoplotdeals with conventional bioinformatics formats (.bed, .fasta, .gff, .gtf, .vcf, ...) to avoid file manipulations before drawing.How does it work ?karyoplottakes as an argument a configuration file with different sections which will have a sequence Fasta file as reference for ideograms. Every plot to draw on reference sequences is described in a track section with link to associated data and drawing options.karyoplotis mainly based onmatplotlibfor graph rendering and pysam for bioinformatics file format parsing.karyoplotis under developpement, currently supported format are:fastabedavailable soon:gffvcfbamwig/bigWigInstallcreate your virtual env in a dedicated directorypython3 -m venv $HOME/myvenvs/karyoplotset up the new envsource $HOME/myvenvs/karyoplot/bin/activateinstall karyoplotpip install karyoplotenjoykaryoplot -hto leave the envdeactivateUsageSimple usagekaryoplot config.ini -o genome_karyo.png -v 2Theconfig.inifile is a configuration file in Microsoft Windows INI style, readable by theconfigparserPython module.
Some sections are expected with defined options to customize your plot. See the associated documentationherefor full options and required sections/options.SupportFor any request please contact nicolas.lapalu[at]inrae.fr or adeline.simon[at]inrae.frAuthors and acknowledgmentRosanne PhebeNicolas LapaluAdeline SimonLicenselicensed under GNU General Public License v3.0References[1] Krzywinski, M. et al. Circos: an Information Aesthetic for Comparative Genomics. Genome Res (2009) 19:1639-1645[2] Bernat Gel & Eduard Serra. (2017). karyoploteR: an R/Bioconductor package to plot customizable genomes displaying arbitrary data. Bioinformatics, 31–33. doi:10.1093/bioinformatics/btx346
Package: karyopype
karyopypeKaryopype is a simple chromosome plotting package in python allowing you to quickly visualize where a set of genomic regions, or multiple sets of genomic regions fall on the chromosomes.From an interactive python session (jupyter notebook):pipinstallkaryopypeimportkaryopype.karyopypeaskp# plot two sets of regionskp.plot_karyopype("hg38",["/path/to/regions.bed","/path/to/regions2.bed"])From a command line sessionpipinstallkaryopypepython>>>importkaryopype.karyopypeaskp# save plot to current directory>>>kp.plot_karyopyp("hg38","/path/to/regions.bed",savefig=True)
Package: kas
Build StatusThis tool provides an easy mechanism to setup bitbake based
projects.The OpenEmbedded tooling support starts at step 2 with bitbake. The
downloading of sources and then configuration has to be done by
hand. Usually, this is explained in a README. Instead kas is using a
project configuration file and does the download and configuration
phase.Key features provided by the build tool:clone and checkout bitbake layerscreate default bitbake settings (machine, arch, …)launch minimal build environment, reducing risk of host contaminationinitiate bitbake build processSee thekas documentationfor further details.SECURITY NOTICEAt this stage, kas does not validate the integrity of fetched repositories.
Make sure to only pull from trusted sources to ensure that the selected
revisions are the expected ones, specifically when using mirrors. Later
versions of kas may introduce integrity validation mechanisms such as
cryptographic checksums to strengthen supply chain security.
Package: kasa-crypt
kasa-cryptFast kasa cryptInstallationInstall this via pip (or your favourite package manager):pip install kasa-cryptExample usagefromkasa_cryptimportencrypt,decryptencrypted=encrypt('{"hello":"world"}')decrypted=decrypt(encrypted[4:])# Always a 4 byte headerprint(decrypted)Contributors ✨Thanks goes to these wonderful people (emoji key):This project follows theall-contributorsspecification. Contributions of any kind welcome!CreditsThis package was created withCopierand thebrowniebroke/pypackage-templateproject template.