Transformer Hangup.
Author: dotsinspace
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.42.4
- Platform: Linux-5.4.0-105-generic-x86_64-with-glibc2.31
- Python version: 3.12.4
- Huggingface_hub version: 0.23.5
- Safetensors version: 0.4.3
- Accelerate version: 0.32.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.3.1+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: "Just created the Class wrapped around transformer pipeline."
- RAM 120GB
- Architecture:                    x86_64
- CPU op-mode(s):                  32-bit, 64-bit
- Byte Order:                      Little Endian
- Address sizes:                   40 bits physical, 48 bits virtual
- CPU(s):                          24
- On-line CPU(s) list:             0-23
- Thread(s) per core:              1
- Core(s) per socket:              24
- Socket(s):                       1
- NUMA node(s):                    1
- Vendor ID:                       AuthenticAMD
- CPU family:                      23
- Model:                           49
- Model name:                      AMD EPYC 7282 16-Core Processor
- Stepping:                        0
- CPU MHz:                         2794.748
- BogoMIPS:                        5589.49
- Hypervisor vendor:               KVM
- Virtualization type:             full
- L1d cache:                       1.5 MiB
- L1i cache:                       1.5 MiB
- L2 cache:                        12 MiB
- L3 cache:                        16 MiB
- NUMA node0 CPU(s):               0-23
- Vulnerability Itlb multihit:     Not affected
- Vulnerability L1tf:              Not affected
- Vulnerability Mds:               Not affected
- Vulnerability Meltdown:          Not affected
- Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
- Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
- Vulnerability Spectre v2:        Mitigation; LFENCE, IBPB conditional, STIBP disabled, RSB filling
- Vulnerability Srbds:             Not affected
- Vulnerability Tsx async abort:   Not affected
- Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3
                                  fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd ibpb stibp vmmcall fsgs
                                 base tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 wbnoinvd arat umip arch_capabilities


                                 

### Who can help?

@Narsil @ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
#
# IMPORTS
#
import os  # CORE: OS library for managing the environment.
import torch # PIP: PyTorch library for managing tensors.
import optuna  # PIP: Optuna library for hyperparameter optimization.
import transformers # PIP: Hugging Face Transformers library for managing the model and tokenizer.
from dotsinspace_debugger import Debugger  # PIP: Debugger library for handling errors.
from transformers import AutoModelForCausalLM, AutoTokenizer, DataCollatorForLanguageModeling, Trainer, TrainingArguments # PIP: Hugging Face Transformers library for managing the model and tokenizer.


#
# SIBLINGS
#
from .Dataset import LocalDataset


#
# CLASS
#
#
# IMPORTS
#
import torch  # PIP: PyTorch library for managing tensors.
import transformers  # PIP: Hugging Face Transformers library for managing the model and tokenizer.
from dotsinspace_debugger import Debugger  # PIP: Debugger library for handling errors.
from transformers import AutoTokenizer  # PIP: Hugging Face Transformers library for managing the model and tokenizer.
import asyncio  # CORE: asyncio library for handling asynchronous operations.
import functools  # CORE: functools library for handling functional programming.
from concurrent.futures import ThreadPoolExecutor  # CORE: ThreadPoolExecutor for running blocking code in an asynchronous manner.


#
# CLASS
#
class Groot:
    # Constructor.
    def __init__(self, directory):
        # Check if MPS is available, else use CUDA if available, otherwise use CPU
        if torch.backends.mps.is_available():
            # Set device to MPS
            self.device = torch.device('mps')
        elif torch.cuda.is_available():
            # Set device to CUDA
            self.device = torch.device('cuda')
        else:
            # Set device to CPU
            self.device = torch.device('cpu')

        # Property assignment.
        self.baseModel = 'tiiuae/falcon-40b'
        self.Tokenizer = None
        self.Pipeline = None
        self.Executor = ThreadPoolExecutor()
        
    # Function to asynchronously load the pipeline
    # and tokenizer.
    async def Load(self):
        # Variable assignment.
        functionName = 'Root -> Load'
        Debug = Debugger(name=functionName)

        # Error handling.
        try:
            # Style guide.
            Debug.info(f"Loading root.")

            # Load tokenizer asynchronously.
            self.Tokenizer = await self.AsyncLoadTokenizer(self.baseModel)
            
            # Load pipeline asynchronously.
            self.Pipeline = await self.AsyncLoadPipeline('text-generation', model=self.baseModel, tokenizer=self.Tokenizer, torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto",)

            # Style guide.
            Debug.info(f"Loaded root.")

            # Return success.
            return {
                'message': 'Root has been loaded.',
                'status': 'LOAD_SUCCESSFUL'
            }
        except Exception as error:
            # Report failure.
            raise error

    # Helper function for async pipeline execution
    async def AsyncLoadTokenizer(self, *args, **kwargs):
        # Get the event loop.
        loop = asyncio.get_event_loop()

        # Variable assignment.
        FunctionToolPipeline = functools.partial(AutoTokenizer.from_pretrained, *args, **kwargs)

        # Run the function in a separate thread.
        return await loop.run_in_executor(self.Executor, FunctionToolPipeline)
    
    # Helper function for async pipeline execution
    async def AsyncLoadPipeline(self, *args, **kwargs):
        # Get the event loop.
        loop = asyncio.get_event_loop()

        # Variable assignment.
        FunctionToolPipeline = functools.partial(transformers.pipeline, *args, **kwargs)

        # Run the function in a separate thread.
        return await loop.run_in_executor(self.Executor, FunctionToolPipeline)
    
    # Helper function for async pipeline execution
    async def AsyncExecutePipeline(self, *args, **kwargs):
        # Get the event loop.
        loop = asyncio.get_event_loop()

        # Variable assignment.
        FunctionToolPipeline = functools.partial(self.Pipeline, *args, **kwargs)

        # Run the function in a separate thread.
        return await loop.run_in_executor(self.Executor, FunctionToolPipeline)

    # Function to talk to root asynchronously.
    async def Talk(self, message):
        # Variable assignment.
        functionName = 'Root -> Talk'
        Debug = Debugger(name=functionName)

        # Error handling.
        try:
            # Style guide.
            Debug.info(f"Root is responding.")
            
            # If pipeline does not exist.
            # then load pipeline.
            if not self.Pipeline:
                # Load pipeline.
                await self.Load()

            # Get response from root asynchronously.
            response = await self.AsyncExecutePipeline(message, max_length=200, do_sample=True, top_k=10, num_return_sequences=1, eos_token_id=self.Tokenizer.eos_token_id, truncation=True)
            print(response)
            # Return response.
            return {
                'response': response,
                'status': 'READ_SUCCESSFUL'
            }
        except Exception as error:
            # Report failure.
            raise error

```                               

When Talk function is called it outputs nothing. am i missing something here. + Note i didn't trained it or something this class is broken and fixed around using falcon itself and checking out its response without training.      

### Expected behavior

It should output at least few tokens.

Comments:
Hi @dotsinspace, thanks for raising an issue! 

As this involves custom code, it's a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Sure.