Package: highway-star
highway_starScrap biographies from wikipedia categories and plot their life coursesThe main goal of this project is to retrieve all biographies from a desired wikipedia category, and to plot the life course of those persons
with a sankey diagram. Those data could then be analyzed for social purpose.This project was made in partnership with theLEIRIS.InstallationYou can install the project viapip, or any otherPypipackage manager.pipinstallhighway-starNote : you may need some more packages from spacy for Natural Language Processing. This may cause error during your execution.Please run those commands in your console, or in a python script.pipinstallhttps://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-2.0.0/fr_core_news_sm-2.0.0.tar.gz#egg=fr_core_news_sm==2.0.0python-mspacydownloadfrHow to useScrappingThe function above allows you to scrap biographies from every page of the categories and subcategories crawled by this one.fromhighway_star.scrapping.wikipedia_scraperimportscrap_wikipedia_structure_with_contentcontent=scrap_wikipedia_structure_with_content(root_category="Acteur_français",lang="fr")Let's decompose what this function is doing.Admit that you want all biographies that comes from the wikipedia categoryActeurs_français.The algorithm will get every page link in the orange rectangle, and will store information of everysubcategoryin the red rectangle.Then, it will repeat this process for every subcategory, until there are no category left.For example, in the subcategoryActeur_français_de_cinémaof the categoryActeurs_français,
we still have 1 subcategory, and many new pages to scrap, as shown in the figure just below.Then, when it gets to a page, it will scrap all the content within the tags<spanclass="mw-headline"id="Biographie">Biographie</span>and</h2>In order to select only the content that we have for example in the image just below.The result of this function is a pythondict.You will just have to convert this dictionary to a dataframe using pandas :importpandasaspdpd.DataFrame.from_dict(content)To have an output like thisNote that you have here :page_links: links to the pagespages_names: names of the pagessubcategory: category where the page was foundcontent: the content of the biography that has been scrappedPreprocessingOnce you have retrieved your data, you may need to preprocess them.In order to do that, we have two functions, one simple, and the other more complex.Easy but not custom wayfromhighway_star.preprocessing.biography_preprocessorimportsent_to_wordssent_to_words(biographies_column=dataframe_with_biographies["biographies"])The result of this will be a pythonlistof tokenized biographies.just add it to your dataframe usingcontent["biographies_tokenized"]=sent_to_words(biographies_column=dataframe_with_biographies["biographies"])Complex but custom wayNote : To run this function, make sure to install following packagespipinstallhttps://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-2.0.0/fr_core_news_sm-2.0.0.tar.gz#egg=fr_core_news_sm==2.0.0
python-mspacydownloadfrfromhighway_star.preprocessing.biography_preprocessorimportremove_stop_words_from_biographiesremove_stop_words_from_biographies(biographies_column=dataframe_with_biographies["biographies"],custom_stop_words=["ajouter","oui","être","avoir"],use_lemmatization=True,allowed_postags=['NOUN','VERB'])This function does the tokenization, but also :allows you to choose customstop wordsfilter biographies withstop wordsof the packagespacy.load('fr_core_news_sm')allows you to use or notlemmatizationallows you to filter biographies byparts of speech(e.g., 'NOUN', 'VERB').Default instantiation of this function isfromhighway_star.preprocessing.biography_preprocessorimportremove_stop_words_from_biographiesremove_stop_words_from_biographies(biographies_column=dataframe_with_biographies["biographies"])With non filled parameters set to default :custom_stop_words=Noneuse_lemmatization=Falseallowed_postags=NoneVisualizingThe visualization is done using Sankey Diagram, and the algorithmprefixspanPrefixspanPrefixspan is an algorithm of Data Mining that retrieve the most frequent patterns in a set of data.It was developed in 2001 by Pei, Han et. al, inMining Sequential Patterns Efficiently by Prefix-Projected Pattern Growth.It can be implemented in python using the Pypi libraryprefixspan.Considering that a set of data is a set of biographies, it will retrieve most frequent patterns in our biographies.We can manage the length of the patterns it will search.The more your length pattern is high, the more you have a chance that those pattern globe all biographies from the start to the end, but you may not have many patterns.Sankey DiagramThose are great data visualization tools to plot relational data.An implementation could be found in javascript usingHighcharts.fromhighway_star.visualizing.visualizerimportgive_sankey_data_from_prefixspangive_sankey_data_from_prefixspan(dataframe_with_biographies["content_tokenized"],prefixspan_minlen=15,prefixspan_topk=100)This implementation will find the top100patterns of size15.The basic implementation of this function is :fromhighway_star.visualizing.visualizerimportgive_sankey_data_from_prefixspangive_sankey_data_from_prefixspan(dataframe_with_biographies["content_tokenized"])with :prefixspan_minlen=10prefixspan_topk=50The output of this function is already preprocessed prefixspan output for the sankey diagram.It will count the number of relation couples of items have.e.g. in :born Alabama write song buy house
born Alabama buy house
born Europe write song buy houseborn - Alabama =2buy - house =3write - song =2Note that :Alabama - houseis not a valid item, because the two items are not next to each others.Then, execute :fromhighway_star.visualizing.visualizerimportsankey_diagram_with_prefixspan_outputsankey_diagram_with_prefixspan_output(sankey_data_from_prefixspan=sankey_data_from_prefixspan,js_filename="women",html_filename="women",title="Life course of Women French Actress")Where :sankey_data_from_prefixspan: the output of the previous functiongive_sankey_data_from_prefixspanjs_filename: name of the js filehtml_filename: name of the html filetitle: title of the chartDefault implementation is :fromhighway_star.visualizing.visualizerimportsankey_diagram_with_prefixspan_outputsankey_diagram_with_prefixspan_output(sankey_data_from_prefixspan=sankey_data_from_prefixspan)Where :js_filename=datahtml_filename=pagetitle=NoneThis will save locally two files. A html, and a Javascript.Data of the functiongive_sankey_data_from_prefixspanis stocked into the Javascript file.You will just have to open thehtmlfile to discover your plot.