Init

README.md

@@ -1,3 +1,35 @@
 ---
+language: en
+
 license: apache-2.0
 ---
+
+## ELECTRA for IF
+
+**ELECTRA** is a method for self-supervised language representation learning. They are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf).
+
+For a detailed description and experimental results, please refer to the original paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://openreview.net/pdf?id=r1xMH1BtvB).
+
+This repository contains a small ELECTRA discriminator finetuned on a corpus of interactive fiction commands labelled with the WordNet synset offset of the verb in the sentence. The original dataset has been collected from the list of action in the walkthroughs for the game included in the [Jericho](https://github.com/microsoft/jericho) framework and manually annotated. For more information visit https://github.com/aporporato/electra and https://github.com/aporporato/jericho-corpora.
+
+## How to use the discriminator in `transformers`
+
+```python
+from transformers import ElectraForPreTraining, ElectraTokenizerFast
+import torch
+
+discriminator = ElectraForPreTraining.from_pretrained("google/electra-small-discriminator")
+tokenizer = ElectraTokenizerFast.from_pretrained("google/electra-small-discriminator")
+
+sentence = "The quick brown fox jumps over the lazy dog"
+fake_sentence = "The quick brown fox fake over the lazy dog"
+
+fake_tokens = tokenizer.tokenize(fake_sentence)
+fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt")
+discriminator_outputs = discriminator(fake_inputs)
+predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2)
+
+[print("%7s" % token, end="") for token in fake_tokens]
+
+[print("%7s" % int(prediction), end="") for prediction in predictions.squeeze().tolist()]
+```

config.json

@@ -0,0 +1,31 @@
+{
+  "_name_or_path": "wn_full-trainer",
+  "architectures": [
+    "ElectraModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "embedding_size": 128,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 1024,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "electra",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "summary_activation": "gelu",
+  "summary_last_dropout": 0.1,
+  "summary_type": "first",
+  "summary_use_proj": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.17.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a613fb2aa3569bac1a4a5cb2be88706cdccd33a72dddfc78051353e6ec07cb46
+size 54011377

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56a0bfbd98a2fb0c1a57ad0c8a4dd6234cc17b2e27a21b5c33971a5045d5ecd7
+size 54198792

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "google/electra-small-discriminator", "tokenizer_class": "ElectraTokenizer"}