Package: goose3
IntroGoose was originally an article extractor written in Java that has most
recently (Aug2011) been converted to ascala project.This is a complete rewrite in Python. The aim of the software is to
take any news article or article-type web page and not only extract what
is the main body of the article but also all meta data and most probable
image candidate.Goose will try to extract the following information:Main text of an articleMain image of articleAny YouTube/Vimeo movies embedded in articleMeta DescriptionMeta tagsThe Python version was originally rewritten by:Xavier GrangierLicensingIf you find Goose useful or have issues please drop me a line. I’d love
to hear how you’re using it or what features should be improved.Goose is licensed by Gravity.com under the Apache 2.0 license; see the
LICENSE file for more details.On-line DocumentationOn-line documentation is available onRead the Docswhich contains more in-depth
documentation.SetupTo install using pip, with all supported languages, which will install additional dependencies:pip install goose3[all]To install the minimal version:pip install goose3To install just the dependencies for a single language:pip install goose3[chinese]
pip install goose3[arabic]To install from source:mkvirtualenv --no-site-packages goose3
git clone https://github.com/goose3/goose3.git
cd goose3
pip install -r ./requirements/python
python setup.py installTake it for a spin>>>fromgoose3importGoose>>>url='http://edition.cnn.com/2012/02/22/world/europe/uk-occupy-london/index.html?hpt=ieu_c2'>>>g=Goose()>>>article=g.extract(url=url)>>>article.titleu'Occupy London loses eviction fight'>>>article.meta_description"Occupy London protesters who have been camped outside the landmark St. Paul's Cathedral for the past four months lost their court bid to avoid eviction Wednesday in a decision made by London's Court of Appeal.">>>article.cleaned_text[:150](CNN)-OccupyLondonprotesterswhohavebeencampedoutsidethelandmarkSt.Paul's Cathedral for the past four months lost their court bid to avoi>>>article.top_image.srchttp://i2.cdn.turner.com/cnn/dam/assets/111017024308-occupy-london-st-paul-s-cathedral-story-top.jpgConfigurationThere are two ways to pass configuration to goose. The first one is to
pass goose a Configuration() object. The second one is to pass a
configuration dict.For instance, if you want to change the userAgent used by Goose just
pass:>>>g=Goose({'browser_user_agent':'Mozilla'})Switching parsers: Goose can now be used with lxml html parser or lxml
soup parser. By default the html parser is used. If you want to use the
soup parser pass it in the configuration dict :>>>g=Goose({'browser_user_agent':'Mozilla','parser_class':'soup'})One can also set Goose to be more lenient on network exceptions. To turn off
throwing all network exceptions, set the strict configuration setting to false:>>>g=Goose({'strict':False})To turn on image fetching, one can simply enable it using the enable_image_fetching
configuration property:>>>g=Goose({'enable_image_fetching':True})Goose is now language awareFor example, scraping a Spanish content page with correct meta language
tags:>>>fromgoose3importGoose>>>url='http://sociedad.elpais.com/sociedad/2012/10/27/actualidad/1351332873_157836.html'>>>g=Goose()>>>article=g.extract(url=url)>>>article.titleu'Las listas de espera se agravan'>>>article.cleaned_text[:150]u'Los recortes pasan factura a los pacientes. De diciembre de 2010 a junio de 2012 las listas de espera para operarse aumentaron un 125%. Hay m\xe1s ciudad'Some pages don’t have correct meta language tags, you can force it using
configuration :>>>fromgoose3importGoose>>>url='http://www.elmundo.es/elmundo/2012/10/28/espana/1351388909.html'>>>g=Goose({'use_meta_language':False,'target_language':'es'})>>>article=g.extract(url=url)>>>article.cleaned_text[:150]u'Importante golpe a la banda terrorista ETA en Francia. La Guardia Civil ha detenido en un hotel de Macon, a 70 kil\xf3metros de Lyon, a Izaskun Lesaka y 'Passing {‘use_meta_language’: False, ‘target_language’:’es’} will
forcibly select Spanish.Video extraction>>>importgoose3>>>url='http://www.liberation.fr/politiques/2013/08/12/journee-de-jeux-pour-ayrault-dans-les-jardins-de-matignon_924350'>>>g=goose3.Goose({'target_language':'fr'})>>>article=g.extract(url=url)>>>article.movies[<goose.videos.videos.Videoobjectat0x25f60d0>]>>>article.movies[0].src'http://sa.kewego.com/embed/vp/?language_code=fr&playerKey=1764a824c13c&configKey=dcc707ec373f&suffix=&sig=9bc77afb496s&autostart=false'>>>article.movies[0].embed_code'<iframe src="http://sa.kewego.com/embed/vp/?language_code=fr&amp;playerKey=1764a824c13c&amp;configKey=dcc707ec373f&amp;suffix=&amp;sig=9bc77afb496s&amp;autostart=false" frameborder="0" scrolling="no" width="476" height="357"/>'>>>article.movies[0].embed_type'iframe'>>>article.movies[0].width'476'>>>article.movies[0].height'357'Goose in ChineseSome users want to use Goose for Chinese content. Chinese word
segmentation is way more difficult to deal with than occidental
languages. Chinese needs a dedicated StopWord analyser that need to be
passed to the config object.>>>fromgoose3importGoose>>>fromgoose3.textimportStopWordsChinese>>>url='http://www.bbc.co.uk/zhongwen/simp/chinese_news/2012/12/121210_hongkong_politics.shtml'>>>g=Goose({'stopwords_class':StopWordsChinese})>>>article=g.extract(url=url)>>>printarticle.cleaned_text[:150]香港行政长官梁振英在各方压力下就其大宅的违章建筑（僭建）问题到立法会接受质询，并向香港民众道歉。梁振英在星期二（12月10日）的答问大会开始之际在其演说中道歉，但强调他在违章建筑问题上没有隐瞒的意图和动机。一些亲北京阵营议员欢迎梁振英道歉，且认为应能获得香港民众接受，但这些议员也质问梁振英有Goose in ArabicIn order to use Goose in Arabic you have to use the StopWordsArabic
class.>>>fromgoose3importGoose>>>fromgoose3.textimportStopWordsArabic>>>url='http://arabic.cnn.com/2013/middle_east/8/3/syria.clashes/index.html'>>>g=Goose({'stopwords_class':StopWordsArabic})>>>article=g.extract(url=url)>>>printarticle.cleaned_text[:150]دمشق،سوريا(CNN)-أكدتجهاتسوريةمعارضةأنفصائلمسلحةمعارضةلنظامالرئيسبشارالأسدوعلىصلةبـ"الجيش الحر"تمكنتمنالسيطرةعلىمستودعاتللأسلGoose in KoreanIn order to use Goose in Korean you have to use the StopWordsKorean
class.>>>fromgoose3importGoose>>>fromgoose3.textimportStopWordsKorean>>>url='http://news.donga.com/3/all/20131023/58406128/1'>>>g=Goose({'stopwords_class':StopWordsKorean})>>>article=g.extract(url=url)>>>printarticle.cleaned_text[:150]경기도용인에자리잡은민간시험인증전문기업㈜디지털이엠씨(www.digitalemc.com).14년째세계각국의통신·안전·전파규격시험과인증한우물만파고있는이회사박채규대표가만나기로한주인공이다.그는전기전자·무선통신·자동차전장품분야에TODOVideo html5 tag extraction