Package: c42eventextractor
c42eventextractor - Utilities to extract and record Code42 security events and alertsThec42eventextractorpackage provides modules that assist in the retrieval and logging of Code42 security events and
alerts. This is done by exposing handlers that allow developers to supply custom behaviors to occur when retrieving
events. By default, the extractors will simply print their results to stdout, but these handlers can be extended to
allow developers to record the event info to whatever location or format they desire.RequirementsPython 2.7.x or 3.5.0+Code42 Server 6.8.x+py42 version 1.7.0+InstallationInstallc42eventextractorusing pip:You can install thec42eventextractorpackage from PyPI, from source, or from distribution.From PyPIThe easiest and most common way is to usepip:pipinstallc42eventextractorTo install a previous version of c42eventextractor viapip, add the version number. For example, to install version
0.2.9, you would enter:pipinstallc42eventextractor==0.2.9Visit theproject historyon PyPI to see all published versions.From sourceAlternatively, you can install the c42eventextractor package directly fromsource code:gitclonehttps://github.com/code42/c42eventextractor.gitWhen it finishes downloading, from the root project directory, run:pythonsetup.pyinstallFrom distributionIf you want create a.tarball for installing elsewhere, run this command from the project's root directory:pythonsetup.pysdistAfter it finishes building, the.tarball will be located in the newly createddistdirectory. To install it, enter:pipinstallc42eventextractor-[VERSION].tar.gzUsage - Code42 Security EventsTo get all security events, use theFileEventExtractor:fromc42eventextractor.extractorsimportFileEventExtractorfromc42eventextractorimportExtractionHandlersimportpy42.sdkcode42_sdk=py42.sdk.from_local_account("https://example.authority.com","admin@example.com","password",)handlers=ExtractionHandlers()# Add implementations for customizing handling response and getting/setting insertion timestamp cursors:defhandle_response(response):passdefrecord_cursor_position(cursor):passdefget_cursor_position():passhandlers.handle_response=handle_responsehandlers.record_cursor_position=record_cursor_positionhandlers.get_cursor_position=get_cursor_positionextractor=FileEventExtractor(code42_sdk,handlers)extractor.extract()# To get all security events in a particular time range, provide an EventTimestamp filter.# Note that if you use `record_cursor_position`, your event timestamp filter may not apply.frompy42.sdk.queries.fileevents.filtersimportEventTimestamptime_filter=EventTimestamp.in_range(1564694804,1564699999)extractor.extract(time_filter)# If your timestamps are in string format, you can convert them by doing:fromdatetimeimportdatetimebegin_date_string="21 June, 2020"end_date_string="22 June, 2020"begin_date=datetime.strptime(begin_date_string,"%d%B, %Y")end_date=datetime.strptime(end_date_string,"%d%B, %Y")begin_timestamp=(begin_date-datetime.utcfromtimestamp(0)).total_seconds()end_timestamp=(end_date-datetime.utcfromtimestamp(0)).total_seconds()time_filter=EventTimestamp.in_range(begin_timestamp,end_timestamp)extractor.extract(time_filter)# You can put filters in an iterable and unpack them (using the `*` operator) in the `extract()`# method. This is a common use case for programs that need to conditionally build up filters.frompy42.sdk.queries.fileevents.filtersimportDeviceUsername,FilePath_NEEDS_DEVICE_USERNAME_FILTER=False_NEEDS_FILE_PATH_FILTER=Truefilters=[]if_NEEDS_DEVICE_USERNAME_FILTER:filters.append(DeviceUsername.eq("test.user@example.com"))if_NEEDS_FILE_PATH_FILTER:filters.append(FilePath.is("path/to/file"))extractor.extract(*filters)Usage - Code42 Security AlertsGetting alerts is similar to getting security events, use the AlertExtractor with appropriate alert filters from thepy42.sdk.queries.alerts.filtersmodule:fromc42eventextractor.extractorsimportAlertExtractorfrompy42.sdk.queries.alerts.filtersimportAlertState# set up your sdk and handlers hereextractor=AlertExtractor(code42_sdk,handlers)open_filter=AlertState.eq(AlertState.OPEN)extractor.extract(open_filter)Using "OR" queriesThe default behavior of the extractor is to "AND" all filter groups that get passed in for extraction. If you want to
construct an "OR" query between multiple filters, you can set the.use_or_queryattribute on the
extractor instance and it will convert the query to "OR" all filters except those in the.or_query_exempt_filterslist. This list by default contains all of the timestamp filter classes (as using "OR" with a timestamp filter negates any
other filter categories). If you need to exclude any other specific filters from the "OR" group and keep them in the "AND"
part of the query, you can append either the base filter class (which will exempt any instance of that type of query from the
"OR" query), or pass in a constructed filter with value(s) to only exclude that exact filter from the "OR" group.Example to "OR" two filters (FileNameandFileSize) but keep the whole query restricted to only Exposure events:extractor=FileEventExtractor(code42_sdk,handlers)file_name_filter=FileName.eq("document.txt")file_size_filter=FileSize.greater_than(1024*1024*1024)exposure_event_filter=ExposureType.exists()# convert to OR queryextractor.use_or_query=True# make sure the exposure_event_filter is included in the "AND" groupextractor.or_query_exempt_filters.append(exposure_event_filter)extractor.extract(file_name_filter,file_size_filter,exposure_event_filter)OR queries can be done with both theFileEventExtractorandAlertExtractor.HandlersA basic set of handlers is provided in thec42eventextractor.extraction_handlers.ExtractionHandlersclass.These default to printing the response data and any errors to the console and stores cursor position in memory.c42eventextractoralso provides some common logging and formatting implementations that you may find useful for
reporting on security data.For example, to extract and submit file events to a syslog server in CEF format, use the below as yourhandle_responseimplementation:importjsonimportloggingfromc42eventextractor.logging.handlersimportNoPrioritySysLogHandlerfromc42eventextractor.logging.formattersimportFileEventDictToCEFFormattermy_logger=logging.getLogger("MY_LOGGER")handler=NoPrioritySysLogHandler("examplehostname.com")handler.setFormatter(FileEventDictToCEFFormatter())my_logger.addHandler(handler)my_logger.setLevel(logging.INFO)defhandle_response(response):events=json.loads(response.text)["fileEvents"]foreventinevents:my_logger.info(event)To customize processing of results/errors further, or to persist cursor data to a location of your choosing, override
the methods on the provided handlers or create your own handler class with the same method signature asc42eventextractor.extraction_handlers.ExtractionHandlers.Cursor BehaviorBecause extractors automatically check for cursor checkpoints from the provided handlers, if the.extract()method
is called with the same filter classes used to store the checkpoint position (DateObservedfor alerts andInsertionTimestampfor file events), an exception will be raised if a cursor checkpoint already exists, as the
extractor will automatically add its own timestamp filter to the query.CEF Mappingc42eventextractor includes mappings from JSON field names to common event format (CEF). These formatters are available
by importing thec42eventextractor.logging.formattersmodule. To create a logger that logs file events in CEF format
to a file, follow this guide:importloggingfromc42eventextractor.logging.formattersimportFileEventDictToCEFFormatterformatter=FileEventDictToCEFFormatter()handler=logging.FileHandler("output.txt",delay=True,encoding="utf-8")logger=logging.getLogger("extractor_logger")handler.setFormatter(formatter)logger.addHandler(handler)logger.setLevel(logging.INFO)The following tables map the data from JSON field names to CEF as well as forensic search field names.Attribute mappingThe table below maps JSON fields, CEF fields, andForensic Search fieldsto one another.JSON fieldCEF fieldForensic Search fieldactorsuserActorcloudDriveIdaidn/acreateTimestampfileCreateTimeFile Created DatedeviceUiddeviceExternalIdn/adeviceUserNamesuserUsername (Code42)domainNamedvchostFully Qualified Domain NameeventIdexternalIDn/aeventTimestampendDate ObservedexposurereasonExposure TypefileCategoryfileTypeFile CategoryfileNamefnameFilenamefilePathfilePathFile PathfileSizefsizeFile SizeinsertionTimestamprtn/amd5ChecksumfileHashMD5 HashmodifyTimestampfileModificationTimeFile Modified DateosHostNameshostHostnameprocessNamesprocExecutable Name (Browser or Other App)processOwnersprivProcess User (Browser or Other App)publicIpAddresssrcIP Address (public)removableMediaBusTypecs1 (Code42AEDRemovableMediaBusType)Device Bus Type (Removable Media)removableMediaCapacitycn1 (Code42AEDRemovableMediaCapacity)Device Capacity (Removable Media)removableMediaNamecs3 (Code42AEDRemovableMediaName)Device Media Name (Removable Media)removableMediaSerialNumbercs4Device Serial Number (Removable Media)removableMediaVendorcs2 (Code42AEDRemovableMediaVendor)Device Vendor (Removable Media)sharedWithduserShared WithsyncDestinationdestinationServiceNameSync Destination (Cloud)urlfilePathURLuserUidsuidn/awindowTitlerequestClientApplicationTab/Window TitletabUrlrequestTab URLemailSendersuserSenderemailRecipientsduserRecipientsEvent mappingSee the table below to map exfiltration events to CEF signature IDs.Exfiltration eventCEF fieldCREATEDC42200MODIFIEDC42201DELETEDC42202READ_BY_APPC42203EMAILEDC42204