Package: tgscraper
tgscraperThis is a simple package to scrape data from Telegram channels (and other content in the future). It is based on theTelethon.It can be used as a CLI tool (saves scraped data to a CSV file) or as a Python package (returns pandas DataFrame).InstallationInstall to run as a CLI toolTo use it as a CLI, ptobably the best way to install tgscraper is just to clone this repository and run it from there. We use Poetry to manage dependencies and Python version. To install Poetry, follow the instructionshere. After that, run the following commands in the root directory of the project:poetryinstall
poetryaddtgscraperTo run tgscraper as a script, you need Python 3.11 installed(otherwise, Poetry will probably fail to create an environment).Alternatively, you can install dependencies into you own environment without Poetry, usingrequirements.txtfile (it can be found in this repository):pipinstall-rrequirements.txt
pipinstalltgscraperInstall to use in your codeFor poetryIf you use Poetry, just include the dependencies in yourpyproject.tomlfile:[tool.poetry.dependencies]python="^3.11"pydantic="^1.10.4"telethon="^1.26.1"pandas="^1.5.2"Make sure that you don't have any conflicts with the versions of the dependencies.Then, add tgscraper to your project and install it:poetryaddtgscraper
poetryinstallFor other environment managersIf you use other environment managers, you can install tgscraper using pip:pipinstall-rrequirements.txt
pipinstalltgscraperPython versionTo use tgscraper in your code, you need Python 3.11 in the environment that you use.UsageProvide API credentialsTo run tgscraper you need to providetgs_config.tomlfile with your API credentials intgscraperdirectory. You can get them frommy.telegram.org. List of channels you want to scrape is also required here. It looks like this (you can find an example intgs_config.toml.examplefile)[telegram]# api_id = ID HERE (int)# api_hash = HASH HERE (str)# phone = PHONE HERE (str)# username = USERNAME HERE (str)[input]channels=["https://t.me/svtvnews"]# You can include multiple channels hereAlternatively, you can use interactive mode to provide this information (it is enabled automatically if you don't providetgs_config.tomlfile).Run the scraper as a CLI toolWe use Poetry to manage dependencies and build the package. To run the scraper as a CLI tool, you need to run it in Poetry's virtual environment. To do that, run the following command in the root directory of the project:poetryrunpythontgscraper/tgscraper.pyThe script will create a folderoutputin the root directory of the project if it doesn't exist and save the scraped data there. The name of the file will be the name of the channel.Alternatively, you can usepoetry shellto enter the virtual environment and run the scraper just as a regular Python script.Use in your codeAfter pip installing the package, you can use it in your code like this:fromtgscraperimporttgscraperimportpandasaspd# Client a Telegram clientclient=tgscraper.init()# Be aware that you can be prompted to enter your phone number and a code, sent to your Telegram account# Get the posts from the channel into a pandas DataFrameposts_df=tgscraper.get_posts(client,link,limit=100)# Do something with the dataposts_df.to_csv("posts.csv")LoggingBy default, tgscraper saves the logs intologsdirectory, filenames are constructed from date and time. To customize the logging, you can useBasicConfigfromloggingmodule. For example, to log into console, you can use the following code:importlogginglogging.basicConfig(level=logging.INFO)You can also get the logger and customize it:logger=logging.getLogger("tgscraper")# Do something with the loggerContributionFeel free to open an issue or a pull request if you have any suggestions or found a bug.