Package: verispy
verispyLatest ReleaseLicenseThis is a Python package built for working with VERIS data.  This package has two main purposes:Allow the user to extractVERISJSON objects into a Pandas DataFrame structure. The most likely source of VERIS data is the VERIS Community Database (VCDB).Provide some basic data analysis functionality for the DataFrame.  This includes calculating the count and frequency of given enumerations, and plotting a simple horizontal bar chart.InstallationTo install this package, eithergit clonethis repository, andpython-mpipinstall<path>/verispy/Or, you may simply use:pipinstallverispyYou will also need to download the VCDB data:gitclonehttps://github.com/vz-risk/VCDB.gitLoading DataAfter installing, creating a VERIS object is simple. We just need the path to the VCDB json directory:In[1]:fromverispyimportVERISIn[2]:data_dir='../VCDB/data/json/validated/'In[3]:v=VERIS(json_dir=data_dir)We may wish to verify that the VERIS schema URL is correct. Note that the schema comes from theVERISrepo on GitHub. If you cannot connect to the internet every time you runverispy, you can also download the schema and change the path locally with theschema_pathparameter in thejson_to_dffunction.In[4]:v.schema_urlOut[4]:'https://raw.githubusercontent.com/vz-risk/veris/master/verisc-merged.json'Then, we can load the VERIS data from the JSON and assign to a DataFrame:In[5]:veris_df=v.json_to_df(verbose=True)LoadingschemaLoadingJSONfilestoDataFrame.FinishedloadingJSONfilestodataframe.BuildingDataFramewithenumerations.DonebuildingDataFramewithenumerations.Post-ProcessingDataFrame(A4Names,VictimIndustries,Patterns)FinishedbuildingVERISDataFrameInspecting DataThen, we might want to inspect our DataFrame:In[6]:veris_df.shapeOut[6]:(7839,2315)In[7]:veris_df.head()Out[7]:action.Environmentalaction.Error...victim.statevictim.victim_id0FalseFalse...NJC.R.Bard,Inc.1FalseTrue...NaNBritishColumbiaMinistryofFinance2FalseFalse...NaNNaN3FalseFalse...FLCamberwellHighSchool4FalseFalse...NaNLoyalistCertificationServicesExams[5rowsx2315columns]Do a quick value count on one of the enumerations:In[8]:veris_df['action.malware.variety.Ransomware'].value_counts()Out[8]:False7716True123Name:action.malware.variety.Ransomware,dtype:int64Most of the enumerations are True/False values.To see a dictionary of the enumerations, look at theenumerationsattribute in the VERIS object:In[9]:len(v.enumerations)Out[9]:68In[10]:importpprintIn[11]:pprint.pprint(v.enumerations){'action.environmental.variety':['Deterioration','Earthquake','EMI','ESD','Fire','Flood','Hazmat','Humidity','Hurricane','Ice','Landslide','Leak','Lightning','Meteorite','Particulates','Pathogen','Power failure','Temperature','Tornado','Tsunami','Vermin','Volcano','Wind','Other','Unknown'],'action.error.variety':['Capacity shortage','Classification error','Data entry error','Disposal error','Gaffe','Loss',...# many more linesAnalysisTheenum_summary(get enumerations with confidence intervals) function is the main analysis function withinverispy.We can look at top-level enumerations:In[12]:v.enum_summary(veris_df,'action')Out[12]:enumxnfreq0Error22687629.00.297291Hacking20797629.00.272512Misuse16047629.00.210253Physical15177629.00.198854Malware6357629.00.083245Social5177629.00.067776Environmental87629.00.001057Unknown210NaNNaNOr lower-level enumerations:In[13]:v.enum_summary(veris_df,'action.social.variety')Out[13]:enumxnfreq0Phishing350501.00.698601Bribery51501.00.101802Pretexting41501.00.081843Extortion33501.00.065874Forgery16501.00.031945Influence13501.00.025956Other10501.00.019967Baiting2501.00.003998Elicitation2501.00.003999Propaganda2501.00.0039910Scam2501.00.0039911Spam1501.00.0020012Unknown16NaNNaNWe can add a second variable as thebyparameter, and thus get enumerations subsetted by the "by":In[14]:v.enum_summary(veris_df,'action',by='attribute')Out[14]:byenumxnfreq0attribute.AvailabilityPhysical11532350.00.490641attribute.AvailabilityHacking6642350.00.282552attribute.AvailabilityError4462350.00.189793attribute.AvailabilityMalware1382350.00.058724attribute.AvailabilityMisuse672350.00.028515attribute.AvailabilitySocial592350.00.025116attribute.AvailabilityEnvironmental82350.00.003407attribute.AvailabilityUnknown5NaNNaN8attribute.ConfidentialityError22317057.00.316149attribute.ConfidentialityHacking16847057.00.2386310attribute.ConfidentialityMisuse15527057.00.2199211attribute.ConfidentialityPhysical14927057.00.2114212attribute.ConfidentialityMalware5557057.00.0786513attribute.ConfidentialitySocial4597057.00.0650414attribute.ConfidentialityEnvironmental27057.00.0002815attribute.ConfidentialityUnknown198NaNNaN16attribute.IntegrityHacking9161833.00.4997317attribute.IntegrityMalware6351833.00.3464318attribute.IntegritySocial5171833.00.2820519attribute.IntegrityPhysical3211833.00.1751220attribute.IntegrityMisuse2571833.00.1402121attribute.IntegrityError351833.00.0190922attribute.IntegrityEnvironmental01833.00.0000023attribute.IntegrityUnknown15NaNNaNWe can add in a confidence interval by specifying theci_method(currently supported methods:wilson,normal, oragresti_coull, seehttps://www.statsmodels.org/dev/generated/statsmodels.stats.proportion.proportion_confint.htmlfor more information):In[15]:v.enum_summary(veris_df,'action.social.variety',ci_method='wilson')Out[15]:enumxnfreqmethodlowerupper0Phishing350501.00.69860wilson0.657040.737151Bribery51501.00.10180wilson0.078280.131382Pretexting41501.00.08184wilson0.060900.109143Extortion33501.00.06587wilson0.047280.091064Forgery16501.00.03194wilson0.019750.051245Influence13501.00.02595wilson0.015230.043886Other10501.00.01996wilson0.010880.036357Baiting2501.00.00399wilson0.001100.014448Elicitation2501.00.00399wilson0.001100.014449Propaganda2501.00.00399wilson0.001100.0144410Scam2501.00.00399wilson0.001100.0144411Spam1501.00.00200wilson0.000350.0112212Unknown16NaNNaNwilsonNaNNaNAnd we can change the confidence interval width withci_level(default is 0.95):In[16]:v.enum_summary(veris_df,'action.social.variety',ci_method='wilson',ci_level=0.5)Out[16]:enumxnfreqmethodlowerupper0Phishing350501.00.69860wilson0.684600.712241Bribery51501.00.10180wilson0.093040.111272Pretexting41501.00.08184wilson0.073950.090483Extortion33501.00.06587wilson0.058780.073744Forgery16501.00.03194wilson0.027050.037675Influence13501.00.02595wilson0.021570.031196Other10501.00.01996wilson0.016160.024637Baiting2501.00.00399wilson0.002490.006398Elicitation2501.00.00399wilson0.002490.006399Propaganda2501.00.00399wilson0.002490.0063910Scam2501.00.00399wilson0.002490.0063911Spam1501.00.00200wilson0.001030.0038712Unknown16NaNNaNwilsonNaNNaNTheenum_summaryfunction returns a DataFrame. With this enumeration DataFrame, we can then draw a simple horizontal bar chart with theplot_barchartfunction:In[17]:actionci_df=v.enum_summary(veris_df,'action')In[18]:action_fig=v.plot_barchart(actionci_df,'Actions')In[19]:action_fig.show()Clustering with PatternsAnother useful feature of theverispypackage is thedf_to_matrixfunction, which converts the VERIS DataFrame into a matrix of boolean values for selected enumerations. This feature is inspired by the blog postDBIR Data-Driven Coverby Jay Jacobs. This blog post talks about the DBIR "Patterns," which were originally described in the2014 DBIR.Patterns FunctionIn Jay's blog post, he points the reader to a GitHub gist with a function he wrote, getpatternlist.R. We have taken that R function and converted it to Python; it may be found at this gist:dbir_patterns.py.Using this function, we can create a data frame with the DBIR patterns:In[23]:importpandasaspddefget_pattern(df):""" Generates the DBIR "patterns," with liberal inspiration from the getpatternlist.R:https://gist.github.com/jayjacobs/a145cb87551f551fc719Parameters----------df: pd DataFrame with most VERIS encodings already built (from verispy package).Returns-------pd DataFrame with the patterns. Does not return as part of original VERIS DF."""skimmer=df['action.physical.variety.Skimmer']|\(df['action.physical.variety.Tampering']&df['attribute.confidentiality.data.variety.Payment'])espionage=df['actor.external.motive.Espionage']|df['actor.external.variety.State-affiliated']....(morelines,seegist)In[23]:patterns=get_pattern(veris_df)In[24]:patterns['pattern'].value_counts()Out[24]:MiscellaneousErrors1814PrivilegeMisuse1597LostandStolenAssets1460EverythingElse1028WebApplications896PaymentCardSkimmers278Crimeware268Cyber-Espionage248DenialofService162PointofSale88Name:pattern,dtype:int64From here, we can go back to ourveris_dfDataFrame and make a boolean VERIS matrix:In[25]:vmat=v.df_to_matrix(veris_df)In[26]:vmatOut[26]:array([[0,0,0,...,0,0,0],[0,0,0,...,0,0,0],[0,1,1,...,0,1,0],...,[0,0,1,...,0,0,0],[0,0,0,...,0,0,0],[0,0,1,...,0,0,0]])In[27]:vmat.shapeOut[27]:(7839,569)Then, we can do a dimensionality-reducing technique calledTSNE. The following operation may take several minutes:In[28]:fromsklearn.manifoldimportTSNEIn[29]:tsne=TSNE(n_components=2,random_state=42)In[30]:v_tsne=tsne.fit_transform(vmat)Finally, we can create the following plot, which we have colored by DBIR "Pattern", using Seaborn:In[31]:importseabornassnsIn[32]:importpandasaspdIn[33]:importmatplotlib.pyplotaspltIn[34]:tsne_df=pd.DataFrame({'x':v_tsne[:,0],'y':v_tsne[:,1],'pattern':patterns['pattern']})In[35]:tsne_df.head()Out[35]:xypattern00.411892-34.907738PrivilegeMisuse129.37490512.816430MiscellaneousErrors2-63.858070-47.406250Cyber-Espionage3-58.9871067.611073WebApplications4-75.6749277.452817WebApplicationsIn[36]:tsne_centers=tsne_df.groupby(by='pattern').mean()...:tsne_centers['pattern']=tsne_centers.indexIn[37]:p1=sns.lmplot(x='x',y='y',data=tsne_df,fit_reg=False,hue='pattern',...:scatter_kws={'alpha':0.25},size=6)...:In[38]:deflabel_point(df,ax):...:fori,pointindf.iterrows():...:ax.text(point['x']-30,point['y'],point['pattern'])...:In[39]:label_point(tsne_centers,plt.gca())In[40]:plt.show()Unit TestingUnit tested withpytest(veris)verispy$pytest=======================================================testsessionstarts========================================================platformdarwin--Python3.6.5,pytest-3.5.1,py-1.5.3,pluggy-0.6.0
rootdir:/Users/tylerbyers/src/verispy,inifile:
plugins:remotedata-0.2.1,openfiles-0.3.0,doctestplus-0.1.3,arraydiff-0.2
collected8itemsverispy/tests/test_veris.py........[100%]====================================================8passedin11.50seconds=====================================================