loubnabnl HF staff commited on
Commit
8004d8f
1 Parent(s): 8bea96e

update FIM tokens and examples (#3)

Browse files

- update FIM tokens and examples (179cb9ca028ad6daa304ab4d9875bbe7405078c0)

Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -61,7 +61,7 @@ print(tokenizer.decode(outputs[0]))
61
  Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output:
62
 
63
  ```python
64
- input_text = "<fim-prefix>def print_hello_world():\n <fim-suffix>\n print('Hello world!')<fim-middle>"
65
  inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
66
  outputs = model.generate(inputs)
67
  print(tokenizer.decode(outputs[0]))
 
61
  Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output:
62
 
63
  ```python
64
+ input_text = "<fim_prefix>def print_one_two_three():\n print('one')\n <fim_suffix>\n print('three')<fim_middle>"
65
  inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
66
  outputs = model.generate(inputs)
67
  print(tokenizer.decode(outputs[0]))