Repository: getsentry/sentry
Pull request: Add Slack to the list of filtered web crawlers
This adds Slack to the list of filtered web crawlers. Slack lists 3 different user agents for their crawlers and the `Slack` regex covers all 3 (https://api.slack.com/robots).

Filtering Twitter's crawler was suggested in #5284, but this should already be filtered by the generic `bot[\/\s\)\;]` regex. I added tests for this just to ensure that it is actually filtered.

Fixes https://github.com/getsentry/sentry/issues/5284

Problem statement:
Filter Slackbot crawler
We get issues reported in Sentry whenever we post our internal app links in Slack because the Slackbot web crawler tries to load them and fails. Would it be possible to get Slackbot added to the list of crawlers that are ignored in Sentry?

Hints from issue discussion:
+1
The three Slack user agents for their crawlers are detailed at https://api.slack.com/robots.
If we're gonna do this, we might as well also get Facebook and Twitter. Unless we already do. (Not looking at code yet)
Facebook is in the list of crawlers at https://github.com/getsentry/sentry/blob/master/src/sentry/filters/web_crawlers.py
Cool, then let's add Slack and Twitter to this. I can do this at some point unless you want to throw up a PR for it. <3

Commit message: Add Slack and Twitter to the list of filtered web crawlers.

Patch to review:
diff --git a/src/sentry/filters/web_crawlers.py b/src/sentry/filters/web_crawlers.py
index edef1c06bcbfac..a09d18b5ccc37d 100644
--- a/src/sentry/filters/web_crawlers.py
+++ b/src/sentry/filters/web_crawlers.py
@@ -30,6 +30,10 @@
     r'bot[\/\s\)\;]',
     # Generic spider
     r'spider[\/\s\)\;]',
+    # Slack - see https://api.slack.com/robots
+    r'Slack',
+    # Twitter - see https://dev.twitter.com/cards/getting-started#crawling
+    r'Twitterbot',
 )), re.I)
 
 


Code review comments:
diff_hunk: @@ -30,6 +30,10 @@
     r'bot[\/\s\)\;]',
     # Generic spider
     r'spider[\/\s\)\;]',
+    # Slack - see https://api.slack.com/robots
+    r'Slack',
+    # Twitter - see https://dev.twitter.com/cards/getting-started#crawling
+    r'Twitterbot',
path: src/sentry/filters/web_crawlers.py
text: @user1:
So it seems that this is already being covered by L30 above. No?

@author:
Yup I think you're right! I'll get rid of that.