Package: gmane
This project delivers helper classes for the analysis of the GMANE
email database. Install with:$ pip install gmaneor$ python setup.py installFor greater control of customization (and debugging), clone the repo and install with pip with -e:$ git clonehttps://github.com/ttm/gmane.git$ pip install -e <path_to_repo>This install method is especially useful with
reload function from IPython.lib.deepreload and the standard importlib.Functionalities are based on physics articles on interaction networks:
[1] Stability in human interaction networks: primitive typology of vertex, prominence of measures and activity statistics:http://arxiv.org/abs/1310.7769[2] A connective differentiation of textual production in interaction networks:http://arxiv.org/abs/1412.7309[3] Versinus: a visualization method for graphs in evolution:http://arxiv.org/abs/1412.7311With core concepts of 1) analysis of topological structure; 2) analysis of textual production; 3) visualization of evolving structures. Activity distribution along time and among participants are also approached through specific routines and indirectly through 1), 2) and 3).Ideally, this package should ease:
- Downloading GMANE email list data.
- Building elementary data structures with downloaded data.
- Analysis of data through complex networks and NLP criteria.
- Visualization through diverse layout methods.PS.
Implemented measures of symmetry in network agents activity by hand (not found in network and numeric packages) according to [1].Usage exampleDownload messages from one GMANE list:importgmaneasgdl=g.DownloadGmaneData()# saves into ~/.gmane/dl.downloadListsIDS()# acquires all GMANE list_idsdl.downloadListMessages(dl.list_ids[100])dl.cleanDownloadedLists()# remove empty messages for coherencedl.downloadedStats()# creates ~/.gmane/stats.txt# to load message contents to Python objects:# load 10 messages from list with list_id gmane.ietf.rfc822lm=g.LoadMessages("gmane.ietf.rfc822",10)# or access the structures downloaded to your filesystemdl=g.DownloadGmaneData()dl.getDownloadedLists()lms=[]# and download all messages from 5 listsforlist_idindl.downloaded_lists[:5]:lms.append(g.LoadMessages(list_id))# to load first three lists with the greated number# of downloaded messages:dl.downloadedStats()# might take a whileload_msgs=[]forlist_statindl.lists[:3]:list_id=list_stat[0]load_msgs.append(g.LoadMessages(list_id))# to make basic datastructures of a list with# greatest number of messages:ds=g.MessageDataStructures(load_msgs[0])mm=ds.messagesids=ds.message_idsprint("first: ",mm[ids[0]][2],"last:",mm[ids[-1]][2])# circular (directional) statistics for activity along time# (hours of the day, days of the week, days of the month, etc):# mean_vec, mean_angle, size_mean_vec, circular_mean,# circular_variance, circular dispersion# and histogramsts=g.TimeStatistics(ds)print("made overall circular activity statistics along time")# make latex tables to observe distributions within bins of interesthi=100*ts.hours["histogram"]/ts.hours["histogram"].sum()row_labels=list(range(24))tstring=g.parcialSums(row_labels,data=[hi],partials=[1,2,3,4,6,12],partial_labels=["h","2h","3h","4h","6h","12h"],datarow_labels=["APACHE"])g.writeTex(tstring,"here.tex")ps=g.AgentStatistics(ds)print("made overall activity statistics among participants")# build the interaction network of the messages:nw=g.InteractionNetwok(ds)print("number of nodes:{}, number of edges:{}".format(nw.g.number_of_nodes(),nw.g.number_of_edges()))nm=g.NetworkMeasures(nw)# take measures, including symmetry related measuresnp=g.NetworkPartitioning(nm)# partition in primitive typologysa=np.sectorialized_agents# get members of each sectorprint("{}agents in periphery,{}are intermediary and{}hubs".format(sa[0],sa[1],sa[2]))sa=np.sectorialized_agents__# smoothed histogram for classificationprint("{}agents in periphery,{}are intermediary and{}hubs".format(sa[0],sa[1],sa[2]))# drawnd=g.NetworkDrawer()print("drawer started")nd.makeLayout(nm)print("gave (x,y) for each author with 5-15-80")nd2=g.NetworkDrawer()print("drawer two started")nd2.makeLayout(nm,np)print("gave (x,y) for each author with\sectors by comparison with Erdos-Renyi")nd.drawNetwork(iN,nm,"test.png")nd2.drawNetwork(iN,nm,"test2.png")# make basic PCA plots of network measures:npca=g.NetworkPCA(nm)# Plot PCA with a colored primitive sectorsnpca=g.NetworkPCA(nm,np)# Evolves network with measures, partitions,# PCA, principal components and Versinus plots saved to disklm=lms[0]# loaded messages from list with most messagesne=g.NetworkEvolution(step_size=10)ne.evolveRaw(lm.messages,imagerate=4,erdos_sectors=True)# ne.makeVideo() use this to avoid evolving again just to make video# see testDrawer.py or g.NetworkEvolution to make movies:# https://www.youtube.com/watch?v=iS8NwEy291g# after making network evolution measurements and video,# you can both make music:em=g.EvolutionMusic()print("music is done")# avconv -i mixY.wav -i evo[..<depends on the evolution done>..].avi final.avi# delivers you the final.avi animation with a soundtrack relative to network measures# currently it is the 'four hubs dance' by default:# https://www.youtube.com/watch?v=YxDiwzAUPeU# and further analysis of measures and Erdos sectors:et=g.EvolutionTimelines()print("Written png files with network measures along evolution timeline")# Enjoy!Further documentation is in tests/ folder and object docstrings.