Package: advertools
AnnouncingData Science with Python for SEO course: Cohort based course, interactive, live-coding.advertools: productivity & analysis tools to scale your online marketingA digital marketer is a data scientist.Your job is to manage, manipulate, visualize, communicate, understand,
and make decisions based on data.You might be doing basic stuff, like copying and pasting text on spread
sheets, you might be running large scale automated platforms with
sophisticated algorithms, or somewhere in between. In any case your job
is all about working with data.As a data scientist you don’t spend most of your time producing cool
visualizations or finding great insights. The majority of your time is spent
wrangling with URLs, figuring out how to stitch together two tables, hoping
that the dates, won’t break, without you knowing, or trying to generate the
next 124,538 keywords for an upcoming campaign, by the end of the week!advertoolsis a Python package that can hopefully make that part of your job a little easier.Installationpython3-mpipinstalladvertoolsPhilosophy/approachIt’s very easy to learn how to use advertools. There are two main reasons for that.First, it is essentially a set of independent functions that you can easily learn and
use. There are no special data structures, or additional learning that you need. With
basic Python, and an understanding of the tasks that these functions help with, you
should be able to pick it up fairly easily. In other words, if you know how to use an
Excel formula, you can easily use any advertools function.The second reason is thatadvertoolsfollows the UNIX philosophy in its design and
approach. Here is one of the various summaries of the UNIX philosophy by Doug McIlroy:Write programs that do one thing and do it well. Write programs to work together.
Write programs to handle text streams, because that is a universal interface.Let’s see how advertools follows that:Do one thing and do it well:Each function in advertools aims for that. There is a
function that just extracts hashtags from a text list, another one to crawl websites,
one to test which URLs are blocked by robots.txt files, and one for downloading XML
sitemaps. Although they are designed to work together as a full pipeline, they can be
run independently in whichever combination or sequence you want.Write programs to work together:Independence does not mean they are unrelated. The
workflows are designed to aid the online marketing practitioner in various steps for
understanding websites, SEO analysis, creating SEM campaigns and others.Programs to handle text streams because that is a universal interface:In Data
Science the most used data structure that can be considered “universal” is the
DataFrame. So, most functions return either a DataFrame or a file that can be read into
one. Once you have it, you have the full power of all other tools like pandas for
further manipulating the data, Plotly for visualization, or any machine learning
library that can more easily handle tabular data.This way it is kept modular as well as flexible and integrated.
As a next step most of these functions are being converted to no-codeinteractive appsfor non-coders, and taking them to the next
level.SEM CampaignsThe most important thing to achieve in SEM is a proper mapping between the
three main elements of a search campaignKeywords(the intention) ->Ads(your promise) ->Landing Pages(your delivery of the promise)
Once you have this done, you can focus on management and analysis. More importantly,
once you know that you can set this up in an easy way, you know you can focus
on more strategic issues. In practical terms you need two main tables to get started:Keywords: You cangenerate keywords(note I didn’t say research)  with thekw_generatefunction.Ads: There are two approaches that you can use:Bottom-up: You can create text ads for a large number of products by simple
replacement of product names, and providing a placeholder in case your text
is too long. Check out thead_createfunction for more details.Top-down: Sometimes you have a long description text that you want to split
into headlines, descriptions and whatever slots you want to split them into.ad_from_stringhelps you accomplish that.Tutorials and additional resourcesGet started withData Science for Digital Marketing and SEO/SEMSetting a full SEM campaignfor DataCamp’s website tutorialProject to practicegenerating SEM keywords with Pythonon DataCampSetting up SEM campaigns on a large scaletutorial on SEMrushVisualtool to generate keywordsonline based on thekw_generatefunctionSEOProbably the most comprehensive online marketing area that is both technical
(crawling, indexing, rendering, redirects, etc.) and non-technical (content
creation, link building, outreach, etc.). Here are some tools that can help
with your SEOSEO crawler:A generic SEO crawler that can be customized, built with Scrapy, & with several
features:Standard SEO elements extracted by default (title, header tags, body text,
status code, response and request headers, etc.)CSS and XPath selectors: You probably have more specific needs in mind, so
you can easily pass any selectors to be extracted in addition to the
standard elements being extractedCustom settings: full access to Scrapy’s settings, allowing you to better
control the crawling behavior (set custom headers, user agent, stop spider
after x pages, seconds, megabytes, save crawl logs, run jobs at intervals
where you can stop and resume your crawls, which is ideal for large crawls
or for continuous monitoring, and many more options)Following links: option to only crawl a set of specified pages or to follow
and discover all pages through linksrobots.txt downloaderA simple downloader of robots.txt files in a DataFrame format, so you can
keep track of changes across crawls if any, and check the rules, sitemaps,
etc.XML Sitemaps downloader / parserAn essential part of any SEO analysis is to check XML sitemaps. This is a
simple function with which you can download one or more sitemaps (by
providing the URL for a robots.txt file, a sitemap file, or a sitemap indexSERP importer and parser for Google & YouTubeConnect to Google’s API and get the search data you want. Multiple search
parameters supported, all in one function call, and all results returned in a
DataFrameTutorials and additional resourcesA visual tool built with theserp_googfunction to getSERP rankings on GoogleA tutorial onanalyzing SERPs on a large scale with Pythonon SEMrushSERP datasets on Kagglefor practicing on different industries and use casesSERP notebooks on Kagglesome examples on how you might tackle such dataContent Analysis with XML Sitemaps and PythonXML dataset examples:news sites,Turkish news sites,Bloomberg newsText & Content Analysis (for SEO & Social Media)URLs, page titles, tweets, video descriptions, comments, hashtags are some
examples of the types of text we deal with.advertoolsprovides a few
options for text analysisWord frequencyCounting words in a text list is one of the most basic and important tasks in
text mining. What is also important is counting those words by taking in
consideration their relative weights in the dataset.word_frequencydoes
just that.URL AnalysisWe all have to handle many thousands of URLs in reports, crawls, social media
extracts, XML sitemaps and so on.url_to_dfconverts your URLs into
easily readable DataFrames.EmojiProduced with one click, extremely expressive, highly diverse (3k+ emoji),
and very popular, it’s important to capture what people are trying to communicate
with emoji. Extracting emoji, get their names, groups, and sub-groups is
possible. The full emoji database is also available for convenience, as well
as anemoji_searchfunction in case you want some ideas for your next
social media or any kind of communicationextract_ functionsThe text that we deal with contains many elements and entities that have
their own special meaning and usage. There is a group of convenience
functions to help in extracting and getting basic statistics about structured
entities in text; emoji, hashtags, mentions, currency, numbers, URLs, questions
and more. You can also provide a special regex for your own needs.StopwordsA list of stopwords in forty different languages to help in text analysis.Tutorial on DataCamp for creating theword_frequencyfunction and
explaining the importance of the difference betweenabsolute and weighted word frequencyText Analysis for Online MarketersAn introductory article on SEMrushSocial MediaIn addition to the text analysis techniques provided, you can also connect to
the Twitter and YouTube data APIs. The main benefits of usingadvertoolsfor this:Handles pagination and request limits: typically every API has a limited
number of results that it returns. You have to handle pagination when you
need more than the limit per request, which you typically do. This is handled
by defaultDataFrame results: APIs send you back data in a formats that need to be
parsed and cleaned so you can more easily start your analysis. This is also
handled automaticallyMultiple requests: in YouTube’s case you might want to request data for the
same query across several countries, languages, channels, etc. You can
specify them all in one request and get the product of all the requests in
one responseTutorials and additional resourcesA visual tool tocheck what is trending on Twitterfor all available locationsATwitter data analysis dashboardwith many optionsHow to use theTwitter data API with PythonExtracting entities from social media poststutorial on KaggleAnalyzing 131k tweetsby European Football clubs tutorial on KaggleAn overview of theYouTube data API with PythonConventionsFunction names mostly start with the object you are working on, so you can use
autocomplete to discover other options:kw_: for keywords-related functionsad_: for ad-related functionsurl_: URL tracking and generationextract_: for extracting entities from social media posts (mentions, hashtags, emoji, etc.)emoji_: emoji related functions and objectstwitter: a module for querying the Twitter API and getting results in a DataFrameyoutube: a module for querying the YouTube Data API and getting results in a DataFrameserp_: get search engine results pages in a DataFrame, currently available: Google and YouTubecrawl: a function you will probably use a lot if you do SEO*_to_df: a set of convenience functions for converting to DataFrames
(log files, XML sitemaps, robots.txt files, and lists of URLs)Change Log - advertools0.14.2 (2024-02-24)ChangedAllowsitemap_to_dfto work on offline sitemaps.0.14.1 (2024-02-21)FixedPreserve the order of supplied URLs in the output ofurl_to_df.0.14.0 (2024-02-18)AddedNew modulecrawlyticsfor analyzing crawl DataFrames. Includes functions to
analyze crawl DataFrames (images,redirects, andlinks), as well as
functions to handle large files (jl_to_parquet,jl_subset,parquet_columns).Newencodingoption forlogs_to_df.Option to save the output ofurl_to_dfto a parquet file.ChangedRemove requirement to delete existing log output and error files if they exist.
The function will now overwrite them if they do.Autothrottling is enabled by default incrawl_headersto minimize being blocked.FixedAlways get absolute path for img src while crawling.Handle NA src attributes when extracting images.Change fillna(method=”ffill”) to ffill forurl_to_df.0.13.5 (2023-08-22)AddedInitial experimental functionality forcrawl_images.ChangedEnable autothrottling by default forcrawl_headers.0.13.4 (2023-07-26)Fixed
-  Make img attributes consistent in length, and support all attributes.0.13.3 (2023-06-27)ChangedAllow optional trailing space in log files (contributed by @andypayne)FixedReplace newlines with spaces while parsing JSON-LD which was causing
errors in some cases.0.13.2 (2022-09-30)AddedCrawling recipe for how to use theDEFAULT_REQUEST_HEADERSto change
the default headers.ChangedSplit long lists of URL while crawling regardless of thefollow_linksparameterFixedClarify that while authenticating for Twitter onlyapp_keyandapp_secretare required, with the option to provideoauth_tokenandoauth_token_secretif/when needed.0.13.1 (2022-05-11)AddedCommand line interface with most functionsMake documentation interactive for most pages usingthebe-sphinxChangedUsenp.nanwherever there are missing values inurl_to_dfFixedDon’t remove double quotes from etags when downloading XML sitemapsReplace instances ofpd.DataFrame.appendwithpd.concat, which is
depracated.Replace empty values with np.nan for the size column inlogs_to_df0.13.0 (2022-02-10)AddedNew functioncrawl_headers: A crawler that only makesHEADrequests
to a known list of URLs.New functionreverse_dns_lookup: A way to get host information for a
large list of IP addresses concurrently.New options for crawling:exclude_url_params,include_url_params,exclude_url_regex, andinclude_url_regexfor controlling which links to
follow while crawling.FixedAnycustom_settingsoptions given to thecrawlfunction that were
defined using a dictionary can now be set without issues. There was an
issue if those options were not strings.ChangedTheskip_url_paramsoption was removed and replaced with the more
versatileexclude_url_params, which accepts eitherTrueor a list
of URL parameters to exclude while following links.0.12.3 (2021-11-27)FixedCrawler stops when provided with bad URLs in list mode.0.12.0,1,2 (2021-11-27)AddedNew functionlogs_to_df: Convert a log file of any non-JSON format
into a pandas DataFrame and save it to aparquetfile. This also
compresses the file to a much smaller size.Crawler extracts all availableimgattributes: ‘alt’, ‘crossorigin’,
‘height’, ‘ismap’, ‘loading’, ‘longdesc’, ‘referrerpolicy’, ‘sizes’,
‘src’, ‘srcset’, ‘usemap’,  and ‘width’ (excluding global HTML attributes
likestyleanddraggable).New parameter for thecrawlfunctionskip_url_params: Defaults to
False, consistent with previous behavior, with the ability to not
follow/crawl links containing any URL parameters.New column forurl_to_df“last_dir”: Extract the value in the last
directory for each of the URLs.ChangedQuery parameter columns inurl_to_dfDataFrame are now sorted by how
full the columns are (the percentage of values that are notNA)0.11.1 (2021-04-09)AddedThenofollowattribute for nav, header, and footer links.FixedTimeout error while downloading robots.txt files.Make extracting nav, header, and footer links consistent with all links.0.11.0 (2021-03-31)AddedNew parameterrecursiveforsitemap_to_dfto control whether or not
to get all sub sitemaps (default), or to only get the current
(sitemapindex) one.New columns forsitemap_to_df:sitemap_size_mb(1 MB = 1,024x1,024 bytes), andsitemap_last_modifiedandetag(if available).Option to request multiple robots.txt files withrobotstxt_to_df.Option to save downloaded robots DataFrame(s) to a file withrobotstxt_to_dfusing the new parameteroutput_file.Two new columns forrobotstxt_to_df:robotstxt_last_modifiedandetag(if available).RaiseValueErrorincrawlifcss_selectorsorxpath_selectorscontain any of the default crawl column headersNew XPath code recipes for custom extraction.New functioncrawllogs_to_dfwhich converts crawl logs to a DataFrame
provided they were saved while using thecrawlfunction.New columns incrawl:viewport,charset, allhheadings
(whichever is available), nav, header and footer links and text, if
available.Crawl errors don’t stop crawling anymore, and the error message is
included in the output file under a newerrorsand/orjsonld_errorscolumn(s).In case of having JSON-LD errors, errors are reported in their respective
column, and the remainder of the page is scraped.ChangedRemoved column prefixresp_meta_from columns containing itRedirect URLs and reasons are separated by ‘@@’ for consistency with
other multiple-value columnsLinks extracted while crawling are not unique any more (all links are
extracted).Emoji data updated with v13.1.Heading tags are scraped even if they are empty, e.g. <h2></h2>.Default user agent for crawling is now advertools/VERSION.FixedHandle sitemap index files that contain links to themselves, with an
error message included in the final DataFrameError in robots.txt files caused by comments preceded by whitespaceZipped robots.txt files causing a parsing issueCrawl issues on some Linux systems when providing a long list of URLsRemovedColumns from thecrawloutput:url_redirected_to,links_fragment0.10.7 (2020-09-18)AddedNew functionknowledge_graphfor querying Google’s APIFastersitemap_to_dfwith threadsNew parametermax_workersforsitemap_to_dfto determine how fast
it could goNew parametercapitalize_adgroupsforkw_generateto determine
whether or not to keep ad groups as is, or set them to title case (the
default)FixedRemove restrictions on the number of URLs provided tocrawl,
assumingfollow_linksis set toFalse(list mode)JSON-LD issue breaking crawls when it’s invalid (now skipped)RemovedDeprecate theyoutube.guide_categories_list(no longer supported by
the API)0.10.6 (2020-06-30)AddedJSON-LD support in crawling. If available on a page, JSON-LD items will
have special columns, and multiple JSON-LD snippets will be numbered for
easy filteringChangedStricter parsing for rel attributes, making sure they are in link
elements as wellDate column names forrobotstxt_to_dfandsitemap_to_dfunified
as “download_date”Numbering OG, Twitter, and JSON-LD where multiple elements are present in
the same page, follows a unified approach: no numbering for the first
element, and numbers start with “1” from the second element on. “element”,
“element_1”, “element_2” etc.0.10.5 (2020-06-14)AddedNew features for thecrawlfunction:Extract canonical tags if availableExtract alternatehrefandhreflangtags if availableOpen Graph data “og:title”, “og:type”, “og:image”, etc.Twitter cards data “twitter:site”, “twitter:title”, etc.FixedMinor fixes torobotstxt_to_df:Allow whitespace in fieldsAllow case-insensitive fieldsChangedcrawlnow only supportsoutput_filewith the extension “.jl”word_frequencydropswtd_freqandrel_valuecolumns ifnum_listis not provided0.10.4 (2020-06-07)AddedNew functionurl_to_df, splitting URLs into their components and to a
DataFrameSlight speed up forrobotstxt_test0.10.3 (2020-06-03)AddedNew functionrobotstxt_test, testing URLs and whether they can be
fetched by certain user-agentsChangedDocumentation main page relayout, grouping of topics, & sidebar captionsVarious documentation clarifications and new tests0.10.2 (2020-05-25)AddedUser-Agent info to requests getting sitemaps and robotstxt filesCSS/XPath selectors support for the crawl functionSupport for custom spider settings with a new parametercustom_settingsFixedUpdate changed supported search operators and values for CSE0.10.1 (2020-05-23)ChangedLinks are better handled, and new output columns are available:links_url,links_text,links_fragment,links_nofollowbody_textextraction is improved by containing <p>, <li>, and <span>
elements0.10.0 (2020-05-21)AddedNew functioncrawlfor crawling and parsing websitesNew functionrobotstxt_to_dfdownloading robots.txt files into
DataFrames0.9.1 (2020-05-19)AddedAbility to specify robots.txt file forsitemap_to_dfAbility to retreive any kind of sitemap (news, video, or images)Errors column to the returnd DataFrame if any errors occurA newsitemap_downloadedcolumn showing datetime of getting the
sitemapFixedLogging issue causingsitemap_to_dfto log the same action twiceIssue preventing URLs not ending with xml or gz from being retreivedCorrect sitemap URL showing in thesitemapcolumn0.9.0 (2020-04-03)AddedNew functionsitemap_to_dfimports an XML sitemap into aDataFrame0.8.1 (2020-02-08)ChangedColumnquery_timeis now namedqueryTimein theyoutubefunctionsHandle json_normalize import from pandas based on pandas version0.8.0 (2020-02-02)AddedNew moduleyoutubeconnecting to all GET requests in APIextract_numbersnew functionemoji_searchnew functionemoji_dfnew variable containing all emoji as a DataFrameChangedEmoji database updated to v13.0serp_googwith expandedpagemapand metadataFixedserp_googerrors, some parameters not appearing in result
dfextract_numbersissue when providing dash as a separator
in the middle0.7.3 (2019-04-17)AddedNew functionextract_exclamationsvery similar toextract_questionsNew functionextract_urls, also counts top domains and
top TLDsNew keys toextract_emoji;top_emoji_categories&top_emoji_sub_categoriesGroups and sub-groups toemoji db0.7.2 (2019-03-29)ChangedEmoji regex updatedSimpler extraction of Spanishquestions0.7.1 (2019-03-26)FixedMissing __init__ imports.0.7.0 (2019-03-26)AddedNewextract_functions:Genericextractused by all others, and takes
arbitrary regex to extract text.extract_questionsto get question mark statistics, as
well as the text of questions asked.extract_currencyshows text that has currency symbols in it, as
well as surrounding text.extract_intense_wordsgets statistics about, and extract words with
any character repeated three or more times, indicating an intense
feeling (+ve or -ve).New functionword_tokenize:Used byword_frequencyto get tokens of
1,2,3-word phrases (or more).Split a list of text into tokens of a specified number of words each.New stop-words from thespaCypackage:current:Arabic, Azerbaijani, Danish, Dutch, English, Finnish,
French, German, Greek, Hungarian, Italian, Kazakh, Nepali, Norwegian,
Portuguese, Romanian, Russian, Spanish, Swedish, Turkish.new:Bengali, Catalan, Chinese, Croatian, Hebrew, Hindi, Indonesian,
Irish, Japanese, Persian, Polish, Sinhala, Tagalog, Tamil, Tatar, Telugu,
Thai, Ukrainian, Urdu, VietnameseChangedword_frequencytakes new parameters:regexdefaults to words, but can be changed to anything ‘S+’
to split words and keep punctuation for example.sepnot longer used as an option, the aboveregexcan
be used insteadnum_listnow optional, and defaults to counts of 1 each if not
provided. Useful for countingabs_freqonly if data not
available.phrase_lenthe number of words in each split token. Defaults
to 1 and can be set to 2 or higher. This helps in analyzing phrases
as opposed to words.Parameters supplied toserp_googappear at the beginning
of the result dfserp_youtubenow containsnextPageTokento make
paginating requests easier0.6.0 (2019-02-11)New functionextract_wordsto extract an arbitrary set of wordsMinor updatesad_from_stringslots argument reflects new text
ad lenghtshashtagregex improved0.5.3 (2019-01-31)Fix minor bugsHandle Twitter search queries with 0 results in final request0.5.2 (2018-12-01)Fix minor bugsProperly handle requests for >50 items (serp_youtube)Rewrite test for _dict_productFix issue with string printing error msg0.5.1 (2018-11-06)Fix minor bugs_dict_product implemented with listsMissing keys in some YouTube responses0.5.0 (2018-11-04)New functionserp_youtubeQuery YouTube API for videos, channels, or playlistsMultiple queries (product of parameters) in one function callReponse looping and merging handled, one DataFrameserp_googreturn Google’s original error messagestwitter responses with entities, get the entities extracted, each in a
separate column0.4.1 (2018-10-13)New functionserp_goog(based on Google CSE)Query Google search and get the result in a DataFrameMake multiple queries / requests in one function callAll responses merged in one DataFrametwitter.get_place_trends results are ranked by town and country0.4.0 (2018-10-08)New Twitter module based on twythonWraps 20+ functions for getting Twitter API dataGets data in a pands DataFrameHandles looping over requests higher than the defaultsTested on Python 3.70.3.0 (2018-08-14)Search engine marketing cheat sheet.New set of extract_ functions with summary stats for each:extract_hashtagsextract_mentionsextract_emojiTests and bug fixes0.2.0 (2018-07-06)New set of kw_<match-type> functions.Full testing and coverage.0.1.0 (2018-07-02)First release on PyPI.Functions available:ad_create: create a text ad place words in placeholdersad_from_string: split a long string to shorter string that fit intogiven slotskw_generate: generate keywords from lists of products and wordsurl_utm_ga: generate a UTM-tagged URL for Google Analytics trackingword_frequency: measure the absolute and weighted frequency of words incollection of documents