Package: cyberspacy
spaCy v2.0extension and pipeline component
for tagging IPs, email addresses, URLs, and Windows commandlines.
Heavily inspired byspacymoji.Installationcyberspacyrequiresspacyv2.0.0 or higher.pippipinstallcyberspacyParsing Windows commandlinesYou can use cyberspacy to tokenize, tag, and normalize Windows command lines from endpoint telemetry.fromcyberspacyimportWindowsCommandlineProcessorprocessor=WindowsCommandlineProcessor()cmd_line=r'"C:\Program Files\MyProgram.exe" /d C:\Users\Alice\file.txt --file C:\test.py'assertprocessor.get_args(cmd_line)==["/d","--file"]assertprocessor.get_paths(cmd_line)==['"C:\\Program Files\\MyProgram.exe"','C:\\Users\\Alice\\file.txt','C:\\test.py']assertprocessor.get_normalized_paths(cmd_line)==['"?pf64\\myprogram.exe"','?usr\\file.txt','?c\\test.py']assertprocessor.normalize(cmd_line)=='"?pf64\\myprogram.exe" /d ?usr\\file.txt --file ?c\\test.py'Tagging documentsImport the component and initialise it with the sharednlpobject (i.e. an
instance ofLanguage), which is used to initialise thePhraseMatcherwith the shared vocab, and create the match patterns. Then add the component
anywhere in your pipeline.importspacyfromspacy.lang.enimportEnglishfromcyberspacyimportIPTaggernlp=English()ip_Tagger=IPTagger(nlp)nlp.add_pipe(ip_Tagger,first=True)doc=nlp(u'This is a sentence which contains 2.3.4.5 as an IP address')assertdoc._.has_ipv4==Trueassertdoc[0]._.is_ipv4==Falseassertdoc[6]._.is_ipv4==Trueassertlen(doc._.ipv4)==1idx,ipv4_token=doc._.ipv4[0]assertidx==6assertipv4_token.text=='2.3.4.5'cyberspacyonly cares about the token text, so you can use it on a blankLanguageinstance (it should work for allavailable languages!), or in
a pipeline with a loaded model.Available attributesThe extension sets attributes on theDoc,SpanandToken. You can
change the attribute names on initialisation of the extension. For more details
on custom components and attributes, see theprocessing pipelines documentation.The attributes provided by the IPTagger class are:Token._.is_ipv4boolWhether the token is an IPv4 address.Doc._.has_ipv4boolWhether the document contains an IPv4 address.Doc._.ipv4list(index, token)tuples of the document’s IPv4 addresses.Span._.has_ipv4boolWhether the span contains IPv4 addresses.Span._.ipv4list(index, token)tuples of the span’s IPv4 addresses.The attributes provided by the URLTagger class are:Token._.is_urlboolWhether the token is a URL.Doc._.has_urlboolWhether the document contains a URL.Doc._.urllist(index, token)tuples of the document’s URLs.Span._.has_urlboolWhether the span contains a URL.Span._.urllist(index, token)tuples of the span’s URLs.The attributes provided by the EmailTagger class are:Token._.is_email_addrboolWhether the token is an email address.Doc._.has_email_addrboolWhether the document contains an email address.Doc._.email_addrlist(index, token)tuples of the document’s email addresses.Span._.has_email_addrboolWhether the span contains an email address.Span._.email_addrlist(index, token)tuples of the span’s email addresses.The attributes provided by the CommandLineTagger class are:Token._.is_pathboolWhether the token is a path.Token._.is_argboolWhether the token is an argument/flag.Token._.is_valboolWhether the token is a value for an argument.Token._.is_cmdboolWhether the token is a nested command.Doc._.normalizestrReturns a normalized version of the commandlineDoc._.has_pathboolWhether the document contains a path.Doc._.pathlist(index, token)tuples of the document’s paths.Doc._.has_argboolWhether the document contains an argument/flag.Doc._.arglist(index, token)tuples of the document’s args.Doc._.has_valboolWhether the document contains a value for an argument.Doc._.vallist(index, token)tuples of the document’s values.Doc._.has_cmdboolWhether the document contains a nested command.Doc._.cmdlist(index, token)tuples of the document’s subcommands.Span._.has_pathboolWhether the span contains a path.Span._.pathlist(index, token)tuples of the span’s paths.Span._.has_argboolWhether the span contains an argument/flag.Span._.arglist(index, token)tuples of the span’s args.Span._.has_valboolWhether the span contains a value for an argument.Span._.vallist(index, token)tuples of the span’s values.Span._.has_cmdboolWhether the span contains a nested command.Span._.cmdlist(index, token)tuples of the span’s subcommands.