Package: pyBrNews
A Brazilian News Website Data Acquisition Library for PythonpyBrNews Project, made with ❤️ by Lucas Rodrigues (@NepZR).The pyBrNews project is a Python 3 library in development for tasks of data acquisition in Brazilian News Websites, capable for extracting news and comments from this platforms and with it's core utilizing therequests-HTMLlibrary.💾 pyBrNews Library is also available for download and install on PyPI!Click here.🇧🇷 Você está lendo a versão em Inglês deste README. Para ler a versão em Português Brasileiro,clique aqui.📲 InstallationUsing Python Package Manager (PIP), from PyPI:pipinstallpyBrNewsUsing Python Package Manager (PIP), from source (GitHub):pipinstallgit+https://github.com/NepZR/pyBrNews.gitBuilding wheel and installing it directly from source (GitHub):gitclonehttps://github.com/NepZR/pyBrNews.git&&cdpyBrNews/pythonsetup.pybdist_wheelpipinstalldist/pyBrNews-x.x.x-py3-none-any.whl--force-reinstallObs.: Replace x.x.x with the version.📰 Websites and capture groups supportedWebsite nameNewsCommentsURLPortal G1✅ Working⌨️ In progressLinkFolha de São Paulo✅ Working✅ WorkingLinkExame✅ Working⚠️ Not supportedLinkMetrópoles⌨️ In progress⌨️ In progressLinkDatabase: using MongoDB (pyMongo), supported since October 28th, 2022. Also supports local File System storage (JSON / CSV) since October 30, 2022.Internal Modules:pyBrNews.config.database.PyBrNewsDBandpyBrNews.config.database.PyBrNewsFSAdditional Info:to use a local file system storage (JSON / CSV), set the parameteruse_database=Falsein the news package crawlers. Example:crawler = pyBrNews.news.g1.G1News(use_database=False). By default, isTrueand uses the MongoDB database from PyBrNewsDB class.⌨️ Available methodsPackagenewsdefparse_news(self,news_urls:List[Union[str,dict]],parse_body:bool=False,save_html:bool=True)->Iterable[dict]:"""Extracts all the data from the article in a given news platform by iterating over a URL list. Yields adictionary containing all the parsed data from the article.Parameters:news_urls (List[str]): A list containing all the URLs or a data dict to be parsed from a given platform.parse_body (bool): Defines if the article body will be extracted.save_html (bool): Defines if the HTML bytes from the article will be extracted.Returns:Iterable[dict]: Dictionary containing all the article parsed data."""defsearch_news(self,keywords:List[str],max_pages:int=-1)->List[Union[str,dict]]:"""Extracts all the data or URLs from the news platform based on the keywords given. Returns a list containing theURLs / data found for the keywords.Parameters:keywords (List[str]): A list containing all the keywords to be searched in the news platform.max_pages (int): Number of pages to have the articles URLs extracted from.If not set, will catch until the last possible.Returns:List[Union[str, dict]]: List containing all the URLs / data found for the keywords."""Packageconfig.databaseClassPyBrNewsDBdefset_connection(self,host:str="localhost",port:int=27017)->None:"""Sets the connection host:port parameters for the MongoDB. By default, uses the standard localhost:27017 forlocal usage.Parameters:host (str): Hostname or address to connect.port (int): Port to be used in the connection."""definsert_data(self,parsed_data:dict)->None:"""Inserts the parsed data from a news article or extracted comment into the DB Backend (MongoDB - pyMongo).Parameters:parsed_data (dict): Dictionary containing the parsed data from a news article or comment.Returns:None: Shows a success message if the insertion occurred normally. If not, shows an error message."""defcheck_duplicates(self,parsed_data:dict)->bool:"""Checks if the parsed data is already in the database and prevents from being duplicatedin the crawler execution.Parameters:parsed_data (dict): Dictionary containing the parsed data from a news article or comment.Returns:bool: True if the given parsed data is already in the database. False if not."""ClassPyBrNewsFSdefset_save_path(self,fs_save_path:str)->None:"""Sets the save path for all the exported data generated by this Class.Example: set_save_path(fs_save_path="/home/ubuntu/newsData/")Parameters:fs_save_path (str): Desired save path directory, ending with a slash."""defto_json(self,parsed_data:dict)->None:"""Using the parsed data dictionary from a news article or a comment, export the data as an individual JSON file.Parameters:parsed_data (dict): Dictionary containing the parsed data from a news article or a comment."""defexport_all_data(self,full_data:List[dict])->None:"""By a given list of dictionaries containing the parsed data from news or comments, export in a CSV filecontaining all data.Parameters:full_data (List[dict]): List containing the dictionaries of parsed data."""👨🏻‍💻 Project DeveloperLucas Darlindo Freitas RodriguesData Engineer | Backend Python Dev.LinkedIn (lucasdfr)