Generate method does not return hidden states when "output_hidden_states=True" is passed
Author: freQuensy23-coder
Pull request: no
Labels: name: Generation

### System Info

name: embedings
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - blas=1.0=mkl
  - bzip2=1.0.8=h5eee18b_5
  - ca-certificates=2024.3.11=h06a4308_0
  - cffi=1.16.0=py311h5eee18b_0
  - filelock=3.13.1=py311h06a4308_0
  - fsspec=2023.10.0=py311h06a4308_0
  - gmp=6.2.1=h295c915_3
  - gmpy2=2.1.2=py311hc9b5ff0_0
  - intel-openmp=2023.1.0=hdb19cb5_46306
  - jinja2=3.1.3=py311h06a4308_0
  - ld_impl_linux-64=2.38=h1181459_1
  - libffi=3.4.4=h6a678d5_0
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libprotobuf=3.20.3=he621ea3_0
  - libstdcxx-ng=11.2.0=h1234567_1
  - libuuid=1.41.5=h5eee18b_0
  - markupsafe=2.1.3=py311h5eee18b_0
  - mkl=2023.1.0=h213fc3f_46344
  - mkl-service=2.4.0=py311h5eee18b_1
  - mkl_fft=1.3.8=py311h5eee18b_0
  - mkl_random=1.2.4=py311hdb19cb5_0
  - mpc=1.1.0=h10f8cd9_1
  - mpfr=4.0.2=hb69a4c5_1
  - mpmath=1.3.0=py311h06a4308_0
  - ncurses=6.4=h6a678d5_0
  - networkx=3.1=py311h06a4308_0
  - ninja=1.10.2=h06a4308_5
  - ninja-base=1.10.2=hd09550d_5
  - numpy=1.26.4=py311h08b1b3b_0
  - numpy-base=1.26.4=py311hf175353_0
  - openssl=1.1.1w=h7f8727e_0
  - pip=23.3.1=py311h06a4308_0
  - pycparser=2.21=pyhd3eb1b0_0
  - python=3.11.0=h7a1cb2a_3
  - pytorch=2.2.0=cpu_py311h6d93b4c_0
  - readline=8.2=h5eee18b_0
  - setuptools=68.2.2=py311h06a4308_0
  - sqlite=3.41.2=h5eee18b_0
  - sympy=1.12=py311h06a4308_0
  - tbb=2021.8.0=hdb19cb5_0
  - tk=8.6.12=h1ccaba5_0
  - typing_extensions=4.9.0=py311h06a4308_1
  - wheel=0.41.2=py311h06a4308_0
  - xz=5.4.6=h5eee18b_0
  - zlib=1.2.13=h5eee18b_0
  - pip:
      - aiohttp==3.9.3
      - aiosignal==1.3.1
      - annotated-types==0.6.0
      - anyio==3.7.1
      - argon2-cffi==23.1.0
      - argon2-cffi-bindings==21.2.0
      - arrow==1.3.0
      - asttokens==2.4.1
      - async-lru==2.0.4
      - attrs==23.2.0
      - babel==2.14.0
      - beautifulsoup4==4.12.3
      - bleach==6.1.0
      - certifi==2024.2.2
      - charset-normalizer==3.3.2
      - comm==0.2.2
      - context-processor==2.6.0
      - contourpy==1.2.0
      - cycler==0.12.1
      - dataclasses-json==0.6.4
      - debugpy==1.8.1
      - decorator==5.1.1
      - defusedxml==0.7.1
      - distro==1.9.0
      - executing==2.0.1
      - fastapi==0.103.2
      - fastjsonschema==2.19.1
      - fonttools==4.50.0
      - fqdn==1.5.1
      - frozenlist==1.4.1
      - greenlet==3.0.3
      - h11==0.14.0
      - httpcore==1.0.4
      - httpx==0.27.0
      - huggingface-hub==0.21.4
      - idna==3.6
      - ipykernel==6.29.3
      - ipython==8.22.2
      - ipywidgets==8.1.2
      - isoduration==20.11.0
      - jedi==0.19.1
      - json5==0.9.24
      - jsonpatch==1.33
      - jsonpointer==2.4
      - jsonschema==4.21.1
      - jsonschema-specifications==2023.12.1
      - jupyter==1.0.0
      - jupyter-client==8.6.1
      - jupyter-console==6.6.3
      - jupyter-core==5.7.2
      - jupyter-events==0.10.0
      - jupyter-lsp==2.2.4
      - jupyter-server==2.13.0
      - jupyter-server-terminals==0.5.3
      - jupyterlab==4.1.5
      - jupyterlab-pygments==0.3.0
      - jupyterlab-server==2.25.4
      - jupyterlab-widgets==3.0.10
      - kiwisolver==1.4.5
      - langchain==0.1.12
      - langchain-community==0.0.28
      - langchain-core==0.1.32
      - langchain-text-splitters==0.0.1
      - langsmith==0.1.27
      - llvmlite==0.42.0
      - loguru==0.7.2
      - marshmallow==3.21.1
      - matplotlib==3.8.3
      - matplotlib-inline==0.1.6
      - mistune==3.0.2
      - multidict==6.0.5
      - mypy-extensions==1.0.0
      - nbclient==0.10.0
      - nbconvert==7.16.2
      - nbformat==5.10.3
      - nest-asyncio==1.6.0
      - notebook==7.1.2
      - notebook-shim==0.2.4
      - numba==0.59.1
      - openai==1.14.1
      - orjson==3.9.15
      - overrides==7.7.0
      - packaging==23.2
      - pandas==2.2.1
      - pandocfilters==1.5.1
      - parso==0.8.3
      - pexpect==4.9.0
      - pillow==10.2.0
      - platformdirs==4.2.0
      - plotly==5.20.0
      - prometheus-client==0.20.0
      - prompt-toolkit==3.0.43
      - protobuf==5.26.0
      - psutil==5.9.8
      - ptyprocess==0.7.0
      - pure-eval==0.2.2
      - pydantic==2.6.4
      - pydantic-core==2.16.3
      - pyparsing==3.1.2
      - python-dateutil==2.9.0.post0
      - python-json-logger==2.0.7
      - pytz==2024.1
      - pyyaml==6.0.1
      - pyzmq==25.1.2
      - qtconsole==5.5.1
      - qtpy==2.4.1
      - referencing==0.34.0
      - regex==2023.12.25
      - requests==2.31.0
      - rfc3339-validator==0.1.4
      - rfc3986-validator==0.1.1
      - rpds-py==0.18.0
      - safetensors==0.4.2
      - send2trash==1.8.2
      - six==1.16.0
      - sniffio==1.3.1
      - soupsieve==2.5
      - sqlalchemy==2.0.28
      - stack-data==0.6.3
      - starlette==0.27.0
      - tenacity==8.2.3
      - terminado==0.18.1
      - tiktoken==0.6.0
      - tinycss2==1.2.1
      - tokenizers==0.15.2
      - tornado==6.4
      - tqdm==4.66.2
      - traitlets==5.14.2
      - transformers==4.38.2
      - types-python-dateutil==2.9.0.20240316
      - typing-inspect==0.9.0
      - tzdata==2024.1
      - uri-template==1.3.0
      - urllib3==2.2.1
      - wcwidth==0.2.13
      - webcolors==1.13
      - webencodings==0.5.1
      - websocket-client==1.7.0
      - websockets==12.0
      - widgetsnbextension==4.0.10
      - yarl==1.9.4
prefix: /home/alexeyv3/.conda/envs/embedings

### Who can help?

@gante 
When using the generate method of the transformers library with output_hidden_states=True, the expected behavior, based on the documentation, is to receive the output tokens along with the hidden states of the last layer. However, the actual output does not include the hidden_states, only the generated tokens. This discrepancy between the expected output and the actual output suggests that there might be an issue with the generate method or the documentation needs to be updated.



Code to Reproduce
python
```python
import transformers
model = transformers.AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
tokenizer = transformers.AutoTokenizer.from_pretrained('mistralai/Mistral-7B-Instruct-v0.2')
model.generate(**tokenizer("Hello, how are you?", return_tensors="pt"), max_length=100, output_hidden_states=True)
```

Expected Behavior
The output of the generate method should be a dictionary containing both the output_tokens and the hidden_states of the last layer.

Actual Behavior
The actual output is only the tensor of output_tokens, and the hidden_states are missing. 

Environment
Transformers version: '4.38.2'
Platform: linux
Python version:3 11
Additional Context
This functionality is crucial for my speed imporvement of trl library (https://github.com/huggingface/trl/issues/848). 

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Install the transformers library.
Load the mistralai/Mistral-7B-Instruct-v0.2 model using AutoModelForCausalLM.from_pretrained.
Load the tokenizer using AutoTokenizer.from_pretrained with the same model.
Attempt to generate text with model.generate, passing output_hidden_states=True and other necessary parameters.
Observe that the output does not include hidden_states.

### Expected behavior

{output_tokens: ..., 'hiddn_states': ...}

Comments:
Link to documentation - https://huggingface.co/docs/transformers/en/main_classes/text_generation#transformers.GenerationConfig.output_hidden_states


return_dict_in_generate=True fix this problem