Package: telegramweb
Telegram-newsPython package for automatically fetching and pushing news by Telegram.IntroductionThis is a easy-to-learn, flexible and standardized message fetching and pushing framework, especially forTelegramandTelegram Bot.The target news source can be HTML page, JSON and XML. We also provide customized process for unknown data format.Push the latest news to your channel or group once it happens!Install$pipinstalltelegram-newsOr, you can install by cloning this repository:$gitclonehttps://github.com/ESWZY/telegram-news.git
$cdtelegram-news
$pythonsetup.pyinstallPrepareIt does not need much so that you can run your code anywhere.First, ask@BotFatherfor a bot and bot token. After that, create a publicchannelorgroup, and remember chat id you just named. Do not forget to invite your bot into your channel or group and make it an admin.You also need a SQL database. Any SQL database is OK. Especially, I recommendPostgreSQL.Quick deploy on HerokuClick 👇 button to deploy an example for free. Python environment and PostgreSQL database have been prepared.After deployment, start the worker in "Resources" tab, and then you can see the effect in your channel/group, which contains both SCMP news and Wiki news at same time.UsageThose are 3 examples for you to understand how to use the framework.Basic Exampleimportosfromsqlalchemyimportcreate_enginefromsqlalchemy.ormimportSessionfromtelegram_news.templateimportInfoExtractor,NewsPostman# Three required fields:# Your bot token obtained from @BotFatherbot_token=os.getenv("TOKEN")# Add your bots into a channel as an administratorchannel=os.getenv("CHANNEL")# Your database to store old messages.DATABASE_URL=os.getenv("DATABASE_URL")# Create a database sessionengine=create_engine(DATABASE_URL)db=Session(bind=engine.connect())# The news sourceurl="https://en.wikinews.org/wiki/Main_Page"tag="Wiki News"table_name="wikinews"# Info extractor to process data formatie=InfoExtractor()# Select select element by CSS-based selectorie.set_list_selector('#MainPage_latest_news_text > ul > li')ie.set_title_selector('#firstHeading')ie.set_paragraph_selector('#mw-content-text > div > p:not(p:nth-child(1))')ie.set_time_selector('#mw-content-text > div > p:nth-child(1) > strong')ie.set_source_selector('span.sourceTemplate')# Set a max length for post, Max is 4096ie.max_post_length=2000# News postman to manage sending affairnp=NewsPostman(listURLs=[url,],sendList=[channel,],db=db,tag=tag)np.set_bot_token(bot_token)np.set_extractor(ie)np.set_table_name(table_name)# Start to work!np.poll()Typical results:Advanced Exampleimportosfromsqlalchemyimportcreate_enginefromsqlalchemy.ormimportSessionfromtelegram_news.templateimportInfoExtractor,NewsPostmanbot_token=os.getenv("TOKEN")channel=os.getenv("CHANNEL")DATABASE_URL=os.getenv("DATABASE_URL")engine=create_engine(DATABASE_URL)db=Session(bind=engine.connect())# Above code is as same as the basic example, you can reuse those code directlyurl_2="https://www.cnbeta.com/"tag_2="cnBeta"table_name_2="cnbetanews"ie_2=InfoExtractor()ie_2.set_list_selector('.items-area > div > dl > dt > a')ie_2.set_title_selector('header > h1')# Select many target at same timeie_2.set_paragraph_selector('div.cnbeta-article-body > div.article-summary > p, '# Summary only'div.cnbeta-article-body > div.article-content > p')# Content onlyie_2.set_time_selector('header > div > span:nth-child(1)')ie_2.set_source_selector('header > div > span.source')# Select image to display, then the max length is down to 1024ie_2.set_image_selector('div.cnbeta-article-body > div.article-summary > p img, '# From summary only'div.cnbeta-article-body > div.article-content > p img')# From content onlyie_2.max_post_length=1000np_2=NewsPostman(listURLs=[url_2,],sendList=[channel],tag=tag_2,db=db)np_2.set_extractor(ie_2)np_2.set_table_name(table_name_2)np_2.poll()Typical results:Advanced Example for JSON and XMLThe handle for JSON and XML are quite similar. You can convert XML to JSON by functiontelegram_news.utils.xml_to_json, and useNewsPostmanJSONandInfoExtractorJSON. Or, you can useNewsPostmanXMLandInfoExtractorXMLdirectly.You should use key list to recursively route to the information you want.importhashlibimportjsonimportosfromsqlalchemyimportcreate_enginefromsqlalchemy.ormimportSessionfromtelegram_news.templateimportInfoExtractorJSON,NewsPostmanJSONfromtelegram_news.utilsimportxml_to_jsonbot_token=os.getenv("TOKEN")channel=os.getenv("CHANNEL")DATABASE_URL=os.getenv("DATABASE_URL")engine=create_engine(DATABASE_URL)db=Session(bind=engine.connect())url_3="https://www.scmp.com/rss/91/feed"tag_3="SCMP"table_name_3="scmpnews"ie_3=InfoExtractorJSON()# Pre-process the XML string, convert to JSON stringdeflist_pre_process(text):text=json.loads(xml_to_json(text))returnjson.dumps(text)ie_3.set_list_pre_process_policy(list_pre_process)# Route by key listie_3.set_list_router(['rss','channel','item'])ie_3.set_link_router(['link'])ie_3.set_title_router(['title'])ie_3.set_paragraphs_router(['description'])ie_3.set_time_router(['pubDate'])ie_3.set_source_router(['author'])ie_3.set_image_router(['media:thumbnail','@url'])# Customize ID for news itemdefid_policy(link):returnhashlib.md5(link.encode("utf-8")).hexdigest()ie_3.set_id_policy(id_policy)np_3=NewsPostmanJSON(listURLs=[url_3],sendList=[channel],db=db,tag=tag_3)np_3.set_extractor(ie_3)np_3.set_table_name(table_name_3)np_3.poll()Typical results:Parallel ProgramIf you use the same database and send to the same channel, you can simply joint each part of code block, and callpoll()function simultaneously.An example you can find in our Heroku deploy template repo:https://github.com/ESWZY/telegram-news-getting-started/blob/master/main.pyExample ChannelA Telegram channel ofbasic examplefor English Wikinews:@wikinews_en(in English)A Telegram channel for realtime earthquake warning powered by Telegram-news:@earthquake_alert(in Chinese)TODOHTML item listJSON item listXML item listSend ImageSend VideoSend media groupSend fileSend audioFile sending retryCC as e-mailWebhookUpdate message by message IDDocumentGUIFeedbackFeel free to contact with me if you have any question. Also welcome any contribute.If you build a channel by this, don't forget to share that good news with us!LicenseLicensed under the MIT License.