SagemakerEndpoint model doesn't return full output..only when prompted with langchain
Labels: name: area: embeddings, name: auto:bug

### System Info

infra - sagemaker
model - model deployed with `HuggingFaceModel(...).deploy()`
langchain version - v0.0.233
chain types used - RetrievalQA, load_qa_chain

I have a huggingface model deployed behind a sagemaker endpoint which produces outputs as expected when run prediction against it directly. However, when I initialize it with SagemakerEndpoint class from langchain, it only return two characters and sometimes an empty string. I scoured through the internet and langchain docs for the last couple days and my initialization and chain prompting aspects of the code seem to be in line with the docs guidelines and anecdotal recommendations laid out. 
I think this either a lack integration support for huggingface models deployed with sagemaker or I'm missing something here that's not been written in the docs and examples. Please review and let me know either way. 

### Below code will reproduce the behavior I'm experiencing.

```
from langchain import SagemakerEndpoint
from langchain.llms.sagemaker_endpoint import ContentHandlerBase, LLMContentHandler

endpoint = "xxxxxx-2023-07-14-05-34-901"

parameters = {
    "do_sample": True,
    "top_p": 0.95,
    "temperature": 0.1,
    "max_new_tokens": 256,
    "num_return_sequences": 4,
}
    
class ContentHandler(LLMContentHandler):
        content_type = "application/json"
        accepts = "application/json"
        
        def transform_input(self, prompt: str, model_kwargs: Dict) -> bytes:
            input_str = json.dumps({"inputs": prompt, **model_kwargs})
            return input_str.encode('utf-8')
        
        def transform_output(self, output: bytes) -> str:
            response_json = json.loads(output.read().decode("utf-8"))
            return response_json[0]['generated_text']
        
content_handler = ContentHandler()
        
sm_llm=SagemakerEndpoint(
        endpoint_name=endpoint,
        region_name="us-west-2",
        model_kwargs= parameters,
        content_handler=content_handler,
    )

vectordb = Chroma(persist_directory="db", embedding_function=embedding, collection_name="docs")
retriever = vectordb.as_retriever(search_kwargs={'k':3})
print("retriever: ", retriever)

qa_chain = RetrievalQA.from_chain_type(llm=sm_llm, 
                                  chain_type="stuff", 
                                  retriever=retriever,
                                  return_source_documents=True)
```

#### Here's the output (or lack thereof) from prompting with langchain, in above code. 

```
system_prompt = """
# Your are a helpful and harmless assistant for providing clear and succinct answers to questions."""

question = "What is your purpose?"
query = (system_prompt + "
" + question + "
")
llm_response = qa_chain(query)
print(llm_response['result'])

---------------------------------------------------------------------------
Output:

Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

<some doc context from vectordb goes here - removed due to info sensitivity>

Question: # 
# Your are a helpful and harmless assistant for providing clear and succint answers to questions.
What is your purpose?
Helpful Answer: 
```
As shown above, there's no output for 'Helpful Answer: '. 
#### However when I prompt the model directly with a predictor like below, it returns the full output as expected.

### Model returns the full output with the below code which runs direct prediction against the endpoint.

```
import boto3
from sagemaker.huggingface import HuggingFacePredictor
from sagemaker.session import Session

sm_session = Session(boto_session=boto3.session.Session())

payload = {
    "inputs": "What is your purpose?",
    "parameters": {"max_new_tokens": 256, "do_sample": True}
}

local_llm = HuggingFacePredictor(endpoint, sm_session)
chat = local_llm.predict(data=payload)
result = chat[0]["generated_text"]
print(result)
```

#### Output for direct prediction from the code executed above. 

```
What is your purpose?
You seem important. What is your value? What can you do that makes you unique? What is your unique value?

* You are important to the people who know you best. 
* When you accomplish the things you want to do, you will become valuable to the people who matter to you.
* We value you because you are special
```

As it can be seen above, the model returns an output (not a great one) but the fact is it does when I'm prompted and not when wrapped with SagemakerEndpoint class and prompted with prompted with langchain.

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Expected behavior

SagemakerEndpoint Model returning the full output, similar to how it does when prompted directly with a predictor.