Package: redditcleaner
redditcleanerCleans Reddit Text Data 📜 🧹 🧼 🧽Installationpip install redditcleanerAboutReddit uses some characters in the raw text of comments and submission selftexts that may need to be removed if just the plain natural text is required for NLP/Data Science tasks. This Python module cleans this text data.Usageimportredditcleanertext_raw=<Reddittext>text_cleaned=redditcleaner.clean(text_raw)InputIfReddit'sorPushshift's APIis used to retrieve comments or submissions, the raw comment bodies or submission self texts may look like this:Normal text\n\n**Bold**\n\n*Italic*\n\n[Link](https://fsf.org)\n\n
~~Strike-through~~\n\n`Code`\n\n^(Superscript)
\n\n&gt;!Spoiler!&lt;\n\n# Heading\n\nBullet list:\n\n* Item 1\n* Item 2
\n\nNumbered list:\n\n1. Item 1\n2. Item 2\n\n&gt;Quote\n\n 
Code block\n\nTable:\n\n|Cell 1.1|Cell 1.2|\n|:-|:-|\n|Cell 2.1|Cell 2.2|

\n * Find &amp;#x200B; &gt; "\&gt; the "&gt; hidden\ntext [fsf](http://fsf.org)...
This & that in a normal sentence. "manual quote"These characters stem from (Reddit-specific) Markdown formatting. Seeherehow the first bit looks like on Reddit.OutputThis Python module removes these characters and returns the cleaned text. Using the example above, the output would be:Normal text Bold Italic
Strike-through Code Superscript
Spoiler Heading Bullet list: Item 1 Item 2 
Numbered list: 1. Item 1 2. Item 2 Quote
Code block Table: Cell 1.1 Cell 1.2 Cell 2.1 Cell 2.2

Find the hidden text ... This & that in a normal sentence. "manual quote":warning:Common punctuation, numbers, parentheses, quotation marks, emojis, etc. are deliberately not removed, as this data cleaning task pertains to Reddit-specific characters only. An additional round of data cleaning can be applied manually to clean these common items or apply lowercasing, or whatever else is needed.Advanced UsageThecleanfunction supports optional arguments and it can be used as a lambda to be applied on e.g. Pandas DataFrames.Optional ArgumentsSpecific removals of characters can be disabled with optional arguments passed to thecleanfunction. Everything is on by default, but setting one toFalsedisables it.defclean(text,newline=True,quote=True,bullet_point=True,link=True,strikethrough=True,spoiler=True,code=True,superscript=True,table=True,heading=True)E.g.redditcleaner.clean(text,heading=False)Pandas UsageThis simulates a common format used when retrieving this type of data via the Reddit API:# Put "retrieved" text into Pandas Dataframetest_body_1="\n* Find &amp;#x200B; &gt;\"\\&gt; the\"&gt; hidden\ntext [fsf](http://fsf.org)... This & that in a normal sentence.\"manual quote\""test_body_2="Normal text\n\n**Bold**\n\n*Italic*\n\n[Link](https://fsf.org)\n\n~~Strike-through~~\n\n`Code`\n\n^(Superscript)\n\n&gt;!Spoiler!&lt;\n\n# Heading\n\nBullet list:\n\n* Item 1\n* Item 2\n\nNumbered list:\n\n1. Item 1\n2. Item 2\n\n&gt;Quote\n\nCode block\n\nTable:\n\n|Cell 1.1|Cell 1.2|\n|:-|:-|\n|Cell 2.1|Cell 2.2|"importpandasaspddf=pd.DataFrame([['asdf','test_a',test_body_1],['fdsa','test_b',test_body_2]],columns=list(['id','author','body']))# Prepare redditcleanerimportredditcleaner# Apply (map) the function on all body column entriesdf['body']=df['body'].map(redditcleaner.clean)dfidauthorbody0asdftestaFind    the  hidden text ... This & that in a normal sentence. "manual quote"1fdsatestbNormal text  Bold  Italic    Strike-through  Code  Superscript  Spoiler   Heading  Bullet list:   Item 1  Item 2  Numbered list:  1. Item 1 2. Item 2  Quote      Code block  Table:   Cell 1.1 Cell 1.2       Cell 2.1 Cell 2.2ContributingIf I missed any characters that should also be removed, please let me know or feel free to create a PR yourself! :heart: