Package: threatexchange
python-threatexchangeA Python Library to simplify the exchange and use of trust & safety information, especially media hash exchanges. It also contains a CLI calledthreatexchangeto demonstrate the functionality.python-threatexchange is designed to be extensible and comes with a simple model of adding new functionality.To get similar functionality in a deployable service, check out hasher-matcher-actioner.Run the CLI in Docker containerA Dockerfile is provided which allows you to run the CLI with minimal dependencies.First build the container:$ docker build --tag threatexchange .Then run:$ docker run threatexchangeTo persist the configuration and data between invocations, mount the/var/lib/threatexchangevolume:$ docker run --volume $HOME/.threatexchange:/var/lib/threatexchangeInstallationIf you don't havepip, learn how to install ithere.$python3-mpipinstallthreatexchange--upgradeIntroductionTrust and safety is a generally hard problem. An issue that makes the problem harder is that most platforms attempt to keep their platforms safe on their own, despite bad actors and viral content spreading from platform to platform. This results not only in duplicate effort in building out technical capability to detect harmful content, but also duplicate effort in preventing the spread of known harmful content, since each platform is fighting potentially the same copies of content on their own.One technique that can allow platforms to combine efforts to combat harm is by sharing signatures of content that they have already detected on their own, or the inputs to various trust and safety tools that can be used to find harmful content. The most well-known are photo/video hash sharing programs like those operated by the National Center for Exploited and Missing Children (NCMEC) and the Global Internet Forum to Counter Terrorism, Southwest Grid for Learning’s StopNCII.org and Meta's ThreatExchange platform.The python-threatexchange library aims to simplify the exchange of signals via platforms like the above, as well as provide a baseline of functionality available to simplify the testing and creation of new exchanges and techniques, as well as provide cross-compatibility.Philosophy of the LibraryThis library is maintained by a small team at Meta with a limited range of experience, and so we will prioritize the use cases we are most familiar with. We believe that accessibility is a barrier for many platforms and so will put as much as we can in the open. We also understand that it may not make sense to use only publicly visible approaches, and welcome platform-specific modifications and derivatives. However, we also accept pull requests! If you think functionality is widely applicable, or you have a bug bothering you, we accept pull requests! If you are thinking a larger change may be needed (such as adding an entirely new subcommand to the CLI), we appreciate if you reach out to talk through a feature before submitting it!Key ConceptsBelow is a quick overview of the key concepts. If you dig deeper into the library, there are additional considerations that might apply if you are creating your own extensions.The basic flow of data through the system is:Configure which sources of data (signals) you want to pull from (aka collaborations)Download from all sourcesStore the signals and build an efficient matching datastructure (index)Match content against stored signalsCollaborationsA collaboration represents a single collection of data from a single API. This often ties to practical usage such as "A1 video hashes from the NCMEC industry database" or "terrorism photo hashes from Meta's ThreatExchange only from specific applications". In cases where a platform may want to test or take different actions on matching data from one location, Collaborations provide a way to do so.SignalType, Signals, IndicesA SignalType is the encapsulation of a technique that can be used to classify or detect content and the settings for detecting that content can be shared between platforms.A serialization of data that can be used as an input to detect/match content is called a "signal", and this library enforces that every signal be representable has a python str class.SignalType enforces that you provide "naive" or brute force versions of the techniques that can be used for correctness testing. By default, python-threatexchange will use a simple linear scan against these brute force methods. If there are more efficient methods for scanning large datasets, the SignalTypeIndex interface provides a place to store a more complex scaled technique.ContentType, ContentSignalTypes are usually not globally applicable, targeting only a specific type of content such as text, images, or URLs. Additionally, some types of content can be decomposed or processed to extract additional content. Take for example a URL to a post on a social media site with an embedded video hosted on a third site with a description and thumbnail.URL: www.example.com/post/123
              |
              +-- Text: "Look at this cool video"
              |
              +-- Photo: <thumbnail preview>
              |
              +-- URL: www.content-host.com/321.mp4
                                 |
                                 + File: 321.mp4
                                           |
                                           + Video: <bytes>
                                                       |
                                                       + Images: <frame1>, <frame2>...
                                                       |             |
                                                       |             + Text: <from OCR>
                                                       |
                                                       + Audio: <bytes>
                                                                   |
                                                                   + Text: <computer generated transcript>ContentType is a wrapper around traversing this graph and helping find out which techniques are applicable given a given input. It may make sense to create ContentTypes specific to your platform (such as a post type), or to represent specific combinations of signals. Certain imagery may only be harmful if accompanied by certain text and vice-versa.SignalExchangeAPI, Updates, Checkpoints, FetchedMetadata, StorageA SignalExchangeAPI is a location that allows for the exchange of Signals. It's not expected that that every SignalExchangeAPI supports all signals, or that it is hosted by a third party - an API could just be a specific file on disk.The interface defines how a full copy of signals for a single Collaboration can be fetched using sequential, checkpoint-able updates. It also must provide a solution for naive implementations of storage by merging a copy of the data in memory.For some applications, the amount of data will be too large to fit in memory - in that case, a solution that can efficiently merge updates produced by the fetch() function is all that is needed.ExtensionsThis library can make use of extensions provided by any party, public or private, as long as they conform to the conventions established in the library. Extensions are a way to prototype out new techniques, and quickly make them available in existing exchanges. Some exchanges, like ThreatExchange, allow sharing arbitrary data with arbitrary labels, and so a new technique can be rapidly demonstrated cross-platform even if not officially supported.threatexchangeCLIThethreatexchangecli is designed to rapidly demonstrate the value of the library, and if you were in a pinch, could be the basis for an end-to-end solution if needed.UsageWhile the CLI was designed for use with signal exchanges. The normal flow is roughly:configure collaborationsfetch from APIsbuild indicesmatch datacontribute labels and data$threatexchange--help# The help should give a decent overview of functionalities# Step 1: We can skip this step if using the sample data$threatexchangecollabedit...# Step 2: This will save progress, and we'll want to rerun it to get new data periodically$threatexchangefetch# Step 3: This is done by default at the end of step 2, but you can also trigger it manually$threatexchangedataset--rebuild-indices# Step 4: You can match a variety of content and formats$threatexchangematchtext--'bball now?'raw_text-(SampleSignals)INVESTIGATION_SEED
trend_query-(SampleSignals)INVESTIGATION_SEED# You can also debug matching by looking at what hashes are generated:$threatexchangehashvideoexample.mp4
video_md5f09791b743c21f26a189c33b798b8e46# Step 5: Contribute labels$threatexchangelabel...Viewing SignalsTODO$ threatexchange dataset 
$ threatexchange dataset -P --csv > out.csvConnecting to APIs and Getting SignalsA local fileThis is the fastest way to experiment with the CLI functionality and saving contents$ threatexchange hash photo https://github.com/facebook/ThreatExchange/blob/main/pdq/data/misc-images/b.jpg?raw=true
pdq f8f8f0cee0f4a84f06370a22038f63f0b36e2ed596621e1d33e6b39c4e9c9b22
$ threatexchange hash photo https://github.com/facebook/ThreatExchange/blob/main/pdq/data/misc-images/b.jpg?raw=true >> ~/file.txt
$ threatexchange config collab edit local_file --filename ~/file.txt 'file.txt' --create
$ threatexchange fetch
$ threatexchange match photo https://github.com/facebook/ThreatExchange/blob/main/pdq/data/misc-images/b.jpg?raw=true
pdq - (file.txt) INVESTIGATION_SEEDThreatExchangeIf you have access toMeta's ThreatExchange, you can use the library withPrivacyGroupswiththreat_updatesenabled.# Step 1 - configure the default credentials
$ threatexchange config api fb_threat_exchange --access-token '<TOKEN>'
# Step 1 Alternative 1 - TX_ACCESS_TOKEN Environment variable
$ TX_ACCESS_TOKEN='<TOKEN>'
$ export TX_ACCESS_TOKEN

# Step 1 Alternative 2 - ~/.txtoken file
$ touch ~/.txtoken
$ chmod 600 ~/.txtoken
$ echo '<TOKEN>' > ~/.txtoken file

# Step 2 - import configuration
$ threatexchange config api fb_threat_exchange -L
1012185296055235 'Example Collaboration' ...
$ threatexchange config api fb_threat_exchange -I 1012185296055235

# Step 2 Alternative - manually configure via
$ threatexchange config collab edit fb_threatexchange ...

$ threatexchange fetchNCMEC Hash APITheNational Center for Exploited and Missing Children (NCMEC)hosts a number of media hash exchanges related to Child Sexual Abuse Material (CSAM). If you have an account with NCMEC and credentials, you download and use hashes from that API.# Step 1 - configure the default credentials
$ threatexchange config api ncmec --credentials '<USER>' '<PASSWORD>'
# Step 1 Alternative 1 - TX_NCMEC_CREDENTIALS Environment variable
$ TX_NCMEC_CREDENTIALS='<TOKEN>'
$ export TX_NCMEC_CREDENTIALS

# Step 2 - set up config
# Example: NGO database only using esp=1
$ threatexchange config collab edit ncmec --create 'NCMEC NGO' --environment=NGO --only-esp 

$ threatexchange fetchStopNCII.orgStopNCII.orgallows people to upload hashes of intimate imagery/videos when someone is threatening to share them. If you are a partner with credentials, you can download and use hashes from that API.# Step 1 - TX_ACCESS_TOKEN Environment variable - comma separated
$ TX_STOPNCII_KEYS='<FUNCTION_KEY>,<SUBSCRIPTION_KEY>'
$ export TX_STOPNCII_KEYS

# Step 1 Alternative - ~/.tx_stopncii_keys file
$ touch ~/.tx_stopncii_keys
$ chmod 600 ~/.tx_stopncii_keys
$ echo '<FUNCTION_KEY>,<SUBSCRIPTION_KEY>' > ~/.tx_stopncii_keys

# Step 2 - set up config
$ threatexchange config collab edit stop_ncii --create 'StopNCII' 

$ threatexchange fetchAppendixStateThe CLI stores state in~/.threatexchange. There are a few commands which will manipulate this directory, but if you need to factory reset, dorm -r ~/.threatexchangeGeneral Expectation for Compatibility and VersioningWe strive to provide a stable library for use in production systems. To that end, we will use version numbers to help platforms which are using the threatexchange libraries in their own codebase.Public Interfaces:Any API used in extensions (SignalType, ContentType, etc), including their names and paths.Implementations of those APIs in the library (i.e. PDQSignal), (though excluding internal details of those implementations)CLI commands and flagsCLI output format that might be part of a pipeline (ex:threatexchange dataset -Pandthreatexchange matchstdout)CLI statePrivate Interfaces/Internal Details:CLI command implementationsCLI Logging/stderrAny CLI behavior marked as unstable, prototype, or draft in its --helpMajor versions (1.X.X => 2.0.0) Will have breaking changesExtensions (SignalType, ContentType, SignalExchangeAPI) may not be backwards compatibleState (FetchedSignalMetadata, file formats): May not be compatible, but libraries or the CLI may attempt to automatically migrate if possible. Tooling to migrate state may also be available.CLI: Storage formats, commands, may all have changed.Library: Files may be moved or renamedMinor versions (1.0.X => 1.1.X) May change public interfaces, but only in ways that are backwards compatibleExtensions: May gain new methods, or have signatures with new arguments with defaultsState: May be changed only if automatic migration is possible with how the CLI uses it (__setstate__with pickle, TBD with dacite)CLI: Flags may change behavior or move only if previous invocations will do the same thing (i.e. nargs could go from 1 to '*' or '+', or the flag can be renamed if a hidden alias is maintained)Library: Files not in the public interface may be moved or renamed.Revision numbers (1.0.0 => 1.0.1) will be fully backwards compatible.The CLI as an E2E SolutionWhile hasher-matcher-actioner is this repository's attempt at a scaled end-to-end solution, the CLI uses the same libraries and can emulate the same functionality.In order to do that, you'll need to solve a few problems:Storing and potentially distributing config filesCallingthreatexchange fetchperiodicallyDistributing the produced indicesConnecting your content pipeline tothreatexchange matchfrom those indicesRouting matches to your own tooling and infrastructure.Unless you are doing the above on a single machine, your favorite distributed filesystem may handle most of these problems (for example, syncing a single shared ~/.threatexchange directory).