Twitter has shown potential for monitoring public health trends, BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , disease surveillance, BIBREF6 , and providing a rich online forum for cancer patients, BIBREF7 . Social media has been validated as an effective educational and support tool for breast cancer patients, BIBREF8 , as well as for generating awareness, BIBREF9 . Successful supportive organizations use social media sites for patient interaction, public education, and donor outreach, BIBREF10 . The advantages, limitations, and future potential of using social media in healthcare has been thoroughly reviewed, BIBREF11 . Our study aims to investigate tweets mentioning “breast” and “cancer" to analyze patient populations and selectively obtain content relevant to patient treatment experiences.Our previous study, BIBREF0 , collected tweets mentioning “cancer” over several months to investigate the potential for monitoring self-reported patient treatment experiences. Non-relevant tweets (e.g. astrological and horoscope references) were removed and the study identified a sample of 660 tweets from patients who were describing their condition. These self-reported diagnostic indicators allowed for a sentiment analysis of tweets authored by patients. However, this process was tedious, since the samples were hand verified and sifted through multiple keyword searches. Here, we aim to automate this process with machine learning context classifiers in order to build larger sets of patient self-reported outcomes in order to quantify the patent experience.Patients with breast cancer represent a majority of people affected by and living with cancer. As such, it becomes increasingly important to learn from their experiences and understand their journey from their own perspective. The collection and analysis of invisible patient reported outcomes (iPROs) offers a unique opportunity to better understand the patient perspective of care and identify gaps meeting particular patient care needs. Twitter provides a free streaming Application Programming Interface (API), BIBREF12 , for researchers and developers to mine samples of public tweets. Language processing and data mining, BIBREF13 , was conducted using the Python programming language. The free public API allows targeted keyword mining of up to 1% of Twitter's full volume at any given time, referred to as the `Spritzer Feed'. We collected tweets from two distinct Spritzer endpoints from September 15th, 2016 through December 9th, 2017. The primary feed for the analysis collected INLINEFORM0 million tweets containing the keywords `breast' AND `cancer'. See Figure FIGREF2 for detailed Twitter frequency statistics along with the user activity distribution. Our secondary feed searched just for the keyword `cancer' which served as a comparison ( INLINEFORM1 million tweets, see Appendix 1), and helped us collect additional tweets relevant to cancer from patients. The numeric account ID provided in tweets helps to distinguish high frequency tweeting entities.Sentence classification combines natural language processing (NLP) with machine learning to identify trends in sentence structure, BIBREF14 , BIBREF15 . Each tweet is converted to a numeric word vector in order to identify distinguishing features by training an NLP classifier on a validated set of relevant tweets. The classifier acts as a tool to sift through ads, news, and comments not related to patients. Our scheme combines a logistic regression classifier, BIBREF16 , with a Convolutional Neural Network (CNN), BIBREF17 , BIBREF18 , to identify self-reported diagnostic tweets.It is important to be wary of automated accounts (e.g. bots, spam) whose large output of tweets pollute relevant organic content, BIBREF19 , and can distort sentiment analyses, BIBREF20 . Prior to applying sentence classification, we removed tweets containing hyperlinks to remove automated content (some organic content is necessarily lost with this strict constraint).The user tweet distribution in Figure FIGREF2 , shows the number of users as a function of the number of their tweets we collected. With an average frequency of INLINEFORM0 tweets per user, this is a relatively healthy activity distribution. High frequency tweeting accounts are present in the tail, with a single account producing over 12,000 tweets —an automated account served as a support tool called `ClearScan' for patients in recovery. Approximately 98% of the 2.4 million users shared less than 10 posts, which accounted for 70% of all sampled tweets.The Twitter API also provided the number of tweets withheld from our sample, due to rate limiting. Using these overflow statistics, we estimated the sampled proportion of tweets mentioning these keywords. These targeted feeds were able to collect a large sample of all tweets mentioning these terms; approximately 96% of tweets mentioning “breast,cancer” and 65.2% of all tweets mentioning `cancer' while active. More information regarding the types of Twitter endpoints and calculating the sampling proportion of collected tweets is described in Appendix II.Our goal was to analyze content authored only by patients. To help ensure this outcome we removed posts containing a URL for classification, BIBREF19 . Twitter allows users to spread content from other users via `retweets'. We also removed these posts prior to classification to isolate tweets authored by patients. We also accounted for non-relevant astrological content by removing all tweets containing any of the following horoscope indicators: `astrology',`zodiac',`astronomy',`horoscope',`aquarius',`pisces',`aries',`taurus',`leo',`virgo',`libra', and `scorpio'. We preprocessed tweets by lowercasing and removing punctuation. We also only analyzed tweets for which Twitter had identified `en' for the language English. We evaluated tweet sentiments with hedonometrics, BIBREF21 , BIBREF22 , using LabMT, a labeled set of 10,000 frequently occurring words rated on a `happiness' scale by individuals contracted through Amazon Mechanical Turk, a crowd-sourced survey tool. These happiness scores helped quantify the average emotional rating of text by totaling the scores from applicable words and normalizing by their total frequency. Hence, the average happiness score, INLINEFORM0 , of a corpus with INLINEFORM1 words in common with LabMT was computed with the weighted arithmetic mean of each word's frequency, INLINEFORM2 , and associated happiness score, INLINEFORM3 : DISPLAYFORM0 The average happiness of each word was rated on a 9 point scale ranging from extremely negative (e.g., `emergency' 3.06, `hate' 2.34, `die' 1.74) to positive (e.g., `laughter' 8.50, `love' 8.42, `healthy' 8.02). Neutral `stop words' ( INLINEFORM0 , e.g., `of','the', etc.) were removed to enhance the emotional signal of each set of tweets. These high frequency, low sentiment words can dampen a signal, so their removal can help identify hidden trends. One application is to plot INLINEFORM1 as a function of time. The happiness time-series can provide insight driving emotional content in text. In particular, peak and dips (i.e., large deviations from the average) can help identify interesting themes that may be overlooked in the frequency distribution. Calculated scores can give us comparative insight into the context between sets of tweets.“Word shift graphs” introduced in, BIBREF21 , compare the terms contributing to shifts in a computed word happiness from two term frequency distributions. This tool is useful in isolating emotional themes from large sets of text and has been previously validated in monitoring public opinion, BIBREF23 as well as for geographical sentiment comparative analyses, BIBREF24 . See Appendix III for a general description of word shift graphs and how to interpret them.We began by building a validated training set of tweets for our sentence classifier. We compiled the patient tweets verified by, BIBREF0 , to train a logistic regression content relevance classifier using a similar framework as, BIBREF16 . To test the classifier, we compiled over 5 million tweets mentioning the word cancer from a 10% `Gardenhose' random sample of Twitter spanning January through December 2015. See Appendix 1 for a statistical overview of this corpus.We tested a maximum entropy logistic regression classifier using a similar scheme as, BIBREF16 . NLP classifiers operate by converting sentences to word vectors for identifying key characteristics — the vocabulary of the classifier. Within the vocabulary, weights were assigned to each word based upon a frequency statistic. We used the term frequency crossed with the inverse document frequency (tf-idf), as described in , BIBREF16 . The tf-idf weights helped distinguish each term's relative weight across the entire corpus, instead of relying on raw frequency. This statistic dampens highly frequent non-relevant words (e.g. `of', `the', etc.) and enhances relatively rare yet informative terms (e.g. survivor, diagnosed, fighting). This method is commonly implemented in information retrieval for text mining, BIBREF25 . The logistic regression context classifier then performs a binary classification of the tweets we collected from 2015. See Appendix IV for an expanded description of the sentence classification methodology.We validated the logistic model's performance by manually verifying 1,000 tweets that were classified as `relevant'. We uncovered three categories of immediate interest including: tweets authored by patients regarding their condition (21.6%), tweets from friends/family with a direct connection to a patient (21.9%), and survivors in remission (8.8%). We also found users posting diagnostic related inquiries (7.6%) about possible symptoms that could be linked to breast cancer, or were interested in receiving preventative check-ups. The rest (40.2%) were related to `cancer', but not to patients and include public service updates as well as non-patient authored content (e.g., support groups). We note that the classifier was trained on very limited validated data (N=660), which certainly impacted the results. We used this validated annotated set of tweets to train a more sophisticated classifier to uncover self-diagnostic tweets from users describing their personal breast cancer experiences as current patients or survivors.We implemented the Convolutional Neural Network (CNN) with Google's Tensorflow interface, BIBREF26 . We adapted our framework from, BIBREF18 , but instead trained the CNN on these 1000 labeled cancer related tweets. The trained CNN was applied to predict patient self-diagnostic tweets from our breast cancer dataset. The CNN outputs a binary value: positive for a predicted tweet relevant to patients or survivors and negative for these other described categories (patient connected, unrelated, diagnostic inquiry). The Tensorflow CNN interface reported a INLINEFORM0 accuracy when evaluating this set of labels with our trained model. These labels were used to predict self-reported diagnostic tweets relevant to breast cancer patients. A set of 845 breast cancer patient self-diagnostic Twitter profiles was compiled by implementing our logistic model followed by prediction with the trained CNN on 9 months of tweets. The logistic model sifted 4,836 relevant tweets of which 1,331 were predicted to be self-diagnostic by the CNN. Two independent groups annotated the 1,331 tweets to identify patients and evaluate the classifier's results. The raters, showing high inter-rater reliability, individually evaluated each tweet as self-diagnostic of a breast cancer patient or survivor. The rater's independent annotations had a 96% agreement. The classifier correctly identified 1,140 tweets (85.6%) from 845 profiles. A total of 48,113 tweets from these accounts were compiled from both the `cancer' (69%) and `breast' `cancer' (31%) feeds. We provided tweet frequency statistics in Figure FIGREF7 . This is an indicator that this population of breast cancer patients and survivors are actively tweeting about topics related to `cancer' including their experiences and complications.Next, we applied hedonometrics to compare the patient posts with all collected breast cancer tweets. We found that the surveyed patient tweets were less positive than breast cancer reference tweets. In Figure FIGREF8 , the time series plots computed average word happiness at monthly and daily resolutions. The daily happiness scores (small markers) have a high fluctuation, especially within the smaller patient sample (average 100 tweets/day) compared to the reference distribution (average 10,000 tweets/day). The monthly calculations (larger markers) highlight the negative shift in average word happiness between the patients and reference tweets. Large fluctuations in computed word happiness correspond to noteworthy events, including breast cancer awareness month in October, cancer awareness month in February, as well as political debate regarding healthcare beginning in March May and July 2017.In Figure FIGREF9 word shift graphs display the top 50 words responsible for the shift in computed word happiness between distributions. On the left, tweets from patients were compared to all collected breast cancer tweets. Patient tweets, INLINEFORM0 , were less positive ( INLINEFORM1 v. INLINEFORM2 ) than the reference distribution, INLINEFORM3 . There were relatively less positive words `mom', `raise', `awareness', `women', `daughter', `pink', and `life' as well as an increase in the negative words `no(t)', `patients, `dying', `killing', `surgery' `sick', `sucks', and `bill'. Breast cancer awareness month, occurring in October, tends to be a high frequency period with generally more positive and supportive tweets from the general public which may account for some of the negative shift. Notably, there was a relative increase of the positive words `me', `thank', `you' ,'love', and `like' which may indicate that many tweet contexts were from the patient's perspective regarding positive experiences. Many tweets regarding treatment were enthusiastic, supportive, and proactive. Other posts were descriptive: over 165 sampled patient tweets mentioned personal chemo therapy experiences and details regarding their treatment schedule, and side effects. Numerous patients and survivors in our sample had identified their condition in reference to the American healthcare regulation debate. Many sampled views of the proposed legislation were very negative, since repealing the Affordable Care Act without replacement could leave many uninsured. Other tweets mentioned worries regarding insurance premiums and costs for patients and survivors' continued screening. In particular the pre-existing condition mandate was a chief concern of patients/survivors future coverage. This was echoed by 55 of the sampled patients with the hashtag #iamapreexistingcondition (See Table TABREF10 ).Hashtags (#) are terms that categorize topics within posts. In Table TABREF10 , the most frequently occurring hashtags from both the sampled patients (right) and full breast cancer corpus (left). Each entry contains the tweet frequency, number of distinct profiles, and the relative happiness score ( INLINEFORM0 ) for comparisons. Political terms were prevalent in both distributions describing the Affordable Care Act (#aca, #obamacare, #saveaca, #pretectourcare) and the newly introduced American Healthcare Act (#ahca, #trumpcare). A visual representation of these hashtags are displayed using a word-cloud in the Appendix (Figure A4).Tweets referencing the AHCA were markedly more negative than those referencing the ACA. This shift was investigated in Figure FIGREF9 with a word shift graph. We compared American Healthcare Act Tweets, INLINEFORM0 , to posts mentioning the Affordable Care Act, INLINEFORM1 . AHCA were relatively more negative ( INLINEFORM2 v. INLINEFORM3 ) due to an increase of negatively charged words `scared', `lose', `tax', `zombie', `defects', `cut', `depression', `killing', and `worse' . These were references to the bill leaving many patients/survivors without insurance and jeopardizing future treatment options. `Zombie' referenced the bill's potential return for subsequent votes.We have demonstrated the potential of using sentence classification to isolate content authored by breast cancer patients and survivors. Our novel, multi-step sifting algorithm helped us differentiate topics relevant to patients and compare their sentiments to the global online discussion. The hedonometric comparison of frequent hashtags helped identify prominent topics how their sentiments differed. This shows the ambient happiness scores of terms and topics can provide useful information regarding comparative emotionally charged content. This process can be applied to disciplines across health care and beyond.Throughout 2017, Healthcare was identified as a pressing issue causing anguish and fear among the breast cancer community; especially among patients and survivors. During this time frame, US legislation was proposed by Congress that could roll back regulations ensuring coverage for individuals with pre-existing conditions. Many individuals identifying as current breast cancer patients/survivors expressed concerns over future treatment and potential loss of their healthcare coverage. Twitter could provide a useful political outlet for patient populations to connect with legislators and sway political decisions.March 2017 was a relatively negative month due to discussions over American healthcare reform. The American Congress held a vote to repeal the Affordable Care Act (ACA, also referred to as `Obamacare'), which could potentially leave many Americans without healthcare insurance, BIBREF27 . There was an overwhelming sense of apprehension within the `breast cancer' tweet sample. Many patients/survivors in our diagnostic tweet sample identified their condition and how the ACA ensured coverage throughout their treatment.This period featured a notable tweet frequency spike, comparable to the peak during breast cancer awareness month. The burst event peaked on March 23rd and 24th (65k, 57k tweets respectively, see Figure FIGREF2 ). During the peak, 41,983 (34%) posts contained `care' in reference to healthcare, with a viral retweeted meme accounting for 39,183 of these mentions. The tweet read: "The group proposing to cut breast cancer screening, maternity care, and contraceptive coverage." with an embedded photo of a group of predominately male legislators, BIBREF28 . The criticism referenced the absence of female representation in a decision that could deprive many of coverage for breast cancer screenings. The online community condemned the decision to repeal and replace the ACA with the proposed legislation with references to people in treatment who could `die' (n=7,923) without appropriate healthcare insurance coverage. The vote was later postponed and eventually failed, BIBREF29 .Public outcry likely influenced this legal outcome, demonstrating Twitter's innovative potential as a support tool for public lobbying of health benefits. Twitter can further be used to remind, motivate and change individual and population health behavior using messages of encouragement (translated to happiness) or dissatisfaction (translated to diminished happiness), for example, with memes that can have knock on social consequences when they are re-tweeted. Furthermore, Twitter may someday be used to benchmark treatment decisions to align with expressed patient sentiments, and to make or change clinical recommendations based upon the trend histories that evolve with identifiable sources but are entirely in the public domain. Analyzing the fluctuation in average word happiness as well as bursts in the frequency distributions can help identify relevant events for further investigation. These tools helped us extract themes relevant to breast cancer patients in comparison to the global conversation.One area in which Twitter has traditionally fallen short for a communication medium is that of the aural dimension, such as nuances and inflections. However, Twitter now includes pictures, videos and emojis with people revealing or conveying their emotions by use of these communication methods. It is envisaged that the aural and visual dimensions will eventually grow to complement the published text component towards a more refined understanding of feelings, attitudes and health and clinical sentiments.Lack of widespread patient adoption of social media could be a limiting factor to our analysis. A study of breast cancer patients during 2013–2014, BIBREF30 , found social media was a less prominent form of online communication (N = 2578, 12.3%), however with the advent of smartphones and the internet of things (iot) movement, social media may influence a larger proportion of future patients. Another finding noted that online posts were more likely to be positive about their healthcare decision experience or about survivorship. Therefore we cannot at this time concretely draw population-based assumptions from social media sampling. Nevertheless, understanding this online patient community could serve as a valuable tool for healthcare providers and future studies should investigate current social media usage statistics across patients.Because we trained the content classifier with a relatively small corpus, the model likely over-fit on a few particular word embeddings. For example: 'i have stage iv', `i am * survivor', `i had * cancer'. However, this is similar to the process of recursive keyword searches to gather related content. Also, the power of the CNN allows for multiple relative lingual syntax as opposed to searching for static phrases ('i have breast cancer', 'i am a survivor'). The CNN shows great promise in sifting relevant context from large sets of data.Other social forums for patient self reporting and discussion should be incorporated into future studies. For example, as of 2017, https://community.breastcancer.org has built a population of over 199,000 members spanning 145,000 topics. These tools could help connect healthcare professionals with motivated patients. Labeled posts from patients could also help train future context models and help identify adverse symptoms shared among online social communities.Our study focused primarily on English tweets, since this was the language of our diagnostic training sample. Future studies could incorporate other languages using our proposed framework. It would be important to also expand the API queries with translations of `breast' and `cancer'. This could allow for a cross cultural comparison of how social media influences patients and what patients express on social media. We have demonstrated the potential of using context classifiers for identifying diagnostic tweets related to the experience of breast cancer patients. Our framework provides a proof of concept for integrating machine learning with natural language processing as a tool to help connect healthcare providers with patient experiences. These methods can inform the medical community to provide more personalized treatment regimens by evaluating patient satisfaction using social listening. Twitter has also been shown as a useful medium for political support of healthcare policies as well as spreading awareness. Applying these analyses across other social media platforms could provide comparably rich data-sets. For instance, Instagram has been found to contain indicative markers for depression, BIBREF31 . Integrating these applications into our healthcare system could provide a better means of tracking iPROs across treatment regimens and over time.One area in which Twitter has traditionally fallen short for a communication medium is that of the aural dimension, such as nuances and inflections. However, Twitter now includes pictures, videos, and emojis with people revealing or conveying their emotions by use of these communication methods. With augmented reality, virtual reality, and even chatbot interfaces, it is envisaged that the aural and visual dimensions will eventually grow to complement the published text component towards a more refined understanding of feelings, attitudes and health and clinical sentiments.Follow-on studies to our work could be intended to further develop these models and apply them to larger streams of data. Online crowd sourcing tools, like Amazon's Mechanical Turk, implemented in, BIBREF22 , can help compile larger sets of human validated labels to improve context classifiers. These methods can also be integrated into delivering online outreach surveys as another tool for validating healthcare providers. Future models, trained on several thousand labeled tweets for various real world applications should be explored. Invisible patient- reported outcomes should be further investigated via sentiment and context analyses for a better understanding of how to integrate the internet of things with healthcare.Twitter has become a powerful platform for amplifying political voices of individuals. The response of the online breast cancer community to the American Healthcare Act as a replacement to the Affordable Care Act was largely negative due to concerns over loss of coverage. A widespread negative public reaction helped influence this political result. Social media opinion mining could present as a powerful tool for legislators to connect with and learn from their constituents. This can lead to positive impacts on population health and societal well-being. The authors wish to acknowledge the Vermont Advanced Computing Core, which is supported by NASA (NNX-08AO96G) at the University of Vermont which provided High Performance Computing resources that contributed to the research results reported within this poster. EMC was supported by the Vermont Complex Systems Center. CMD and PSD were supported by an NSF BIGDATA grant IIS-1447634. There are three types of endpoints to access data from Twitter. The `spritzer' (1%) and `gardenhose' (10%) endpoints were both implemented to collect publicly posted relevant data for our analysis. The third type of endpoint is the `Firehose' feed, a full 100% sample, which can be purchased via subscription from Twitter. This was unnecessary for our analysis, since our set of keywords yielded a high proportion of the true tweet sample. We quantified the sampled proportion of tweets using overflow statistics provided by Twitter. These `limit tweets', INLINEFORM0 , issue a timestamp along with the approximate number of posts withheld from our collected sample, INLINEFORM1 . The sampling percentage, INLINEFORM2 , of keyword tweets is approximated as the collected tweet total, INLINEFORM3 , as a proportion of itself combined with the sum of the limit counts, each INLINEFORM4 : DISPLAYFORM0 By the end of 2017, Twitter was accumulating an average of 500 million tweets per day, BIBREF32 . Our topics were relatively specific, which allowed us to collect a large sample of tweets. For the singular search term, `cancer', the keyword sampled proportion, INLINEFORM0 , was approximately 65.21% with a sample of 89.2 million tweets. Our separate Twitter spritzer feed searching for keywords `breast AND cancer` OR `lymphedema' rarely surpassed the 1% limit. We calculated a 96.1% sampling proportion while our stream was active (i.e. not accounting for network or power outages). We present the daily overflow limit counts of tweets not appearing in our data-set, and the approximation of the sampling size in Figure A2. Word shift graphs are essential tools for analyzing which terms are affecting the computed average happiness scores between two text distributions, BIBREF33 . The reference word distribution, INLINEFORM0 , serves as a lingual basis to compare with another text, INLINEFORM1 . The top 50 words causing the shift in computed word happiness are displayed along with their relative weight. The arrows ( INLINEFORM2 ) next to each word mark an increase or decrease in the word's frequency. The INLINEFORM3 , INLINEFORM4 , symbols indicate whether the word contributes positively or negatively to the shift in computed average word happiness.In Figure A3, word shift graphs compare tweets mentioning `breast' `cancer' and a random 10% `Gardenhose' sample of non filtered tweets. On the left, `breast',`cancer' tweets were slightly less positive due to an increase in negative words like `fight', `battle', `risk', and `lost'. These distributions had similar average happiness scores, which was in part due to the relatively more positive words `women', mom', `raise', `awareness', `save', `support', and `survivor'. The word shift on the right compares breast cancer patient tweets to non filtered tweets. These were more negative ( INLINEFORM0 = 5.78 v. 6.01) due a relative increase in words like `fighting', `surgery', `against', `dying', `sick', `killing', `radiation', and `hospital'. This tool helped identify words that signal emotional themes and allow us to extract content from large corpora, and identify thematic emotional topics within the data.We built the vocabulary corpus for the logistic model by tokenizing the annotated set of patient tweets by word, removing punctuation, and lowercasing all text. We also included patient unrelated `cancer' tweets collected as a frame of reference to train the classifier. This set of tweets was not annotated, so we made the assumption that tweets not validated by, BIBREF0 were patient unrelated. The proportion, INLINEFORM0 , of unrelated to related tweets has a profound effect on the vocabulary of the logistic model, so we experimented with various ranges of INLINEFORM1 and settled on a 1:10 ratio of patient related to unrelated tweets. We then applied the tf-idf statistic to build the binary classification logistic model.The Tensorflow open source machine learning library has previously shown great promise when applied to NLP benchmark data-sets, BIBREF17 . The CNN loosely works by implementing a filter, called convolution functions, across various subregions of the feature landscape, BIBREF34 , BIBREF35 , in this case the tweet vocabulary. The model tests the robustness of different word embeddings (e.g., phrases) by randomly removing filtered pieces during optimization to find the best predictive terms over the course of training. We divided the input labeled data into training and evaluation to successively test for the best word embedding predictors. The trained model can then be applied for binary classification of text content.

Q: Do the authors report results only on English datasets?

Q: How were breast cancer related posts compiled from the Twitter streaming API?
A: By using  keywords `breast' AND `cancer' in tweet collecting process.

Q: What machine learning and NLP methods were used to sift tweets relevant to breast cancer experiences?
A: ML  logistic regression classifier combined with a Convolutional Neural Network (CNN) to identify self-reported diagnostic tweets.
NLP methods:  tweet conversion to numeric word vector,  removing tweets containing hyperlinks, removing "retweets", removing all tweets containing horoscope indicators,  lowercasing and  removing punctuation.