Upload README.md

README.md

@@ -0,0 +1,160 @@
+---
+language: en
+tags:
+- timelms
+- twitter
+license: mit
+datasets:
+- twitter-api
+---
+
+# Twitter June 2022 (RoBERTa-base, 154M)
+
+This is a RoBERTa-base model trained on 153.86M tweets until the end of June 2022 (15M tweets increment).
+More details and performance scores are available in the [TimeLMs paper](https://arxiv.org/abs/2202.03829).
+
+Below, we provide some usage examples using the standard Transformers interface. For another interface more suited to comparing predictions and perplexity scores between models trained at different temporal intervals, check the [TimeLMs repository](https://github.com/cardiffnlp/timelms).
+
+For other models trained until different periods, check this [table](https://github.com/cardiffnlp/timelms#released-models).
+
+## Preprocess Text 
+Replace usernames and links for placeholders: "@user" and "http".
+If you're interested in retaining verified users which were also retained during training, you may keep the users listed [here](https://github.com/cardiffnlp/timelms/tree/main/data).
+```python
+def preprocess(text):
+    preprocessed_text = []
+    for t in text.split():
+        if len(t) > 1:
+            t = '@user' if t[0] == '@' and t.count('@') == 1 else t
+            t = 'http' if t.startswith('http') else t
+        preprocessed_text.append(t)
+    return ' '.join(preprocessed_text)
+```
+
+## Example Masked Language Model 
+
+```python
+from transformers import pipeline, AutoTokenizer
+
+MODEL = "cardiffnlp/twitter-roberta-base-jun2022-15M-incr"
+fill_mask = pipeline("fill-mask", model=MODEL, tokenizer=MODEL)
+tokenizer = AutoTokenizer.from_pretrained(MODEL)
+
+def pprint(candidates, n):
+    for i in range(n):
+        token = tokenizer.decode(candidates[i]['token'])
+        score = candidates[i]['score']
+        print("%d) %.5f %s" % (i+1, score, token))
+
+texts = [
+    "So glad I'm <mask> vaccinated.",
+    "I keep forgetting to bring a <mask>.",
+    "Looking forward to watching <mask> Game tonight!",
+]
+for text in texts:
+    t = preprocess(text)
+    print(f"{'-'*30}\n{t}")
+    candidates = fill_mask(t)
+    pprint(candidates, 5)
+```
+
+Output: 
+
+```
+------------------------------
+So glad I'm <mask> vaccinated.
+1) 0.48904  not
+2) 0.19832  fully
+3) 0.13791  getting
+4) 0.02852  still
+5) 0.01900  triple
+------------------------------
+I keep forgetting to bring a <mask>.
+1) 0.05997  backpack
+2) 0.05158  charger
+3) 0.05071  book
+4) 0.04741  lighter
+5) 0.03621  bag
+------------------------------
+Looking forward to watching <mask> Game tonight!
+1) 0.54114  the
+2) 0.23145  The
+3) 0.01682  this
+4) 0.01435  Squid
+5) 0.01300  End
+```
+
+## Example Tweet Embeddings
+```python
+from transformers import AutoTokenizer, AutoModel, TFAutoModel
+import numpy as np
+from scipy.spatial.distance import cosine
+from collections import Counter
+
+def get_embedding(text):  # naive approach for demonstration
+  text = preprocess(text)
+  encoded_input = tokenizer(text, return_tensors='pt')
+  features = model(**encoded_input)
+  features = features[0].detach().cpu().numpy() 
+  return np.mean(features[0], axis=0) 
+
+
+MODEL = "cardiffnlp/twitter-roberta-base-jun2022-15M-incr"
+tokenizer = AutoTokenizer.from_pretrained(MODEL)
+model = AutoModel.from_pretrained(MODEL)
+
+query = "The book was awesome"
+tweets = ["I just ordered fried chicken 🐣", 
+          "The movie was great",
+          "What time is the next game?",
+          "Just finished reading 'Embeddings in NLP'"]
+
+sims = Counter()
+for tweet in tweets:
+    sim = 1 - cosine(get_embedding(query), get_embedding(tweet))
+    sims[tweet] = sim
+
+print('Most similar to: ', query)
+print(f"{'-'*30}")
+for idx, (tweet, sim) in enumerate(sims.most_common()):
+    print("%d) %.5f %s" % (idx+1, sim, tweet))
+```
+Output: 
+
+```
+Most similar to:  The book was awesome
+------------------------------
+1) 0.98878 The movie was great
+2) 0.96100 Just finished reading 'Embeddings in NLP'
+3) 0.94927 I just ordered fried chicken 🐣
+4) 0.94668 What time is the next game?
+```
+
+## Example Feature Extraction 
+
+```python
+from transformers import AutoTokenizer, AutoModel, TFAutoModel
+import numpy as np
+
+MODEL = "cardiffnlp/twitter-roberta-base-jun2022-15M-incr"
+tokenizer = AutoTokenizer.from_pretrained(MODEL)
+
+text = "Good night 😊"
+text = preprocess(text)
+
+# Pytorch
+model = AutoModel.from_pretrained(MODEL)
+encoded_input = tokenizer(text, return_tensors='pt')
+features = model(**encoded_input)
+features = features[0].detach().cpu().numpy() 
+features_mean = np.mean(features[0], axis=0) 
+#features_max = np.max(features[0], axis=0)
+
+# # Tensorflow
+# model = TFAutoModel.from_pretrained(MODEL)
+# encoded_input = tokenizer(text, return_tensors='tf')
+# features = model(encoded_input)
+# features = features[0].numpy()
+# features_mean = np.mean(features[0], axis=0) 
+# #features_max = np.max(features[0], axis=0)
+```