Claude v2.1 SQLDatabaseChain produces comments mixed with SQL Statements
Labels: name: auto:bug, name: area: models

### System Info

conda 23.3.1
Python 3.10.6
angchain                 0.0.344                  pypi_0    pypi
langchain-core            0.0.8                    pypi_0    pypi
langchain-experimental    0.0.43                   pypi_0    pypi

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

#This Notebook will show the integration between Generative AI with SAP HANA Cloud
!pip install -q --upgrade langchain boto3 awscli botocore
!pip install -q sqlalchemy-hana langchain_experimental hdbcli
import langchain
from langchain.sql_database import SQLDatabase
from langchain_experimental.sql import SQLDatabaseChain
from langchain.prompts.prompt import PromptTemplate
from langchain.llms.sagemaker_endpoint import LLMContentHandler
from langchain import SagemakerEndpoint
from langchain.llms import bedrock
from urllib.parse import quote  
import sqlalchemy 
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine, select, Table, MetaData, Column, String
import hdbcli
import json

#Next step is to prepare the template for prompt and input to be used by the Generative AI
table_info = "Table Hotel has fields Name,  Address,  City, State, Zip code. \
Table Room has fields Free or Available, Price. \
Table Customer has fields Customer Number, title, first name, name, address, zip code. \
Table Reservation has fields Reservation Number,Arrival Date, Departure Date. \
Table Maintenance has fields Description, Date performed, Performed by."

_DEFAULT_TEMPLATE = """
Given an input question, create a syntactically correct {dialect}  SQL query to run without comments, then provide answer to the question in english based on the result of SQL Query.
Always use schema USER1.  You DO NOT need to check on the SQL Query for common mistake.

{table_info}

Question: {input}"""

PROMPT = PromptTemplate(
    input_variables=["input", "table_info", "dialect"], template=_DEFAULT_TEMPLATE
)

#Next let's instatiate the bedrock and the handler treatment of input and output
class ContentHandler(LLMContentHandler):
    content_type = "application/json"
    accepts = "application/json"

    def transform_input(self, prompt: str, model_kwargs: dict) -> bytes:
        input_str = json.dumps({"text_inputs": prompt, **model_kwargs})
        return input_str.encode('utf-8')
      
    def transform_output(self, output: bytes) -> str:
        response_json = json.loads(output.read().decode("utf-8"))
        return response_json["generated_texts"][0]

content_handler = ContentHandler()

model_parameter = {"temperature": 0, "max_tokens_to_sample": 4000}

#Make sure your account has access to anthropic claude access. This can be enabled from Bedrock console. Access is auto approved.
llm = bedrock.Bedrock(model_id="anthropic.claude-v2:1", model_kwargs=model_parameter, region_name="us-west-2")

#Let's connect to the SAP HANA Database and then execute langchain SQL Database Chain to query from the Generative AI
db = SQLDatabase.from_uri("hana://USER:Password@<mydb>.hana.trial-us10.hanacloud.ondemand.com:443")
db_chain = SQLDatabaseChain.from_llm(llm=llm, db=db, prompt=PROMPT, verbose=True, use_query_checker=True, top_k=5 )

Execute the first query, this is a simple English to text SQL
db_chain.run("How many Hotels are there ?")


UNEXPECTED OUTPUT *****************
> Entering new SQLDatabaseChain chain...
How many Hotels are there ?
SQLQuery:I don't see any issues with the original SQL query. It looks good as written. Here is the query again:

```sql
SELECT COUNT(*) AS num_hotels
FROM hotel;
```

This counts the number of rows in the hotel table to get the number of unique hotels and returns the result in the num_hotels column alias. The query follows proper practices and does not contain any of the common mistakes listed.
---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1819, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1818     if not evt_handled:
-> 1819         self.dialect.do_execute(
   1820             cursor, statement, parameters, context
   1821         )
   1823 if self._has_events or self.engine._has_events:

File /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/default.py:732, in DefaultDialect.do_execute(self, cursor, statement, parameters, context)
    731 def do_execute(self, cursor, statement, parameters, context=None):
--> 732     cursor.execute(statement, parameters)

ProgrammingError: (257, 'sql syntax error: incorrect syntax near "I": line 1 col 1 (at pos 1)')

The above exception was the direct cause of the following exception:

ProgrammingError                          Traceback (most recent call last)
Cell In[43], line 2
      1 #Execute the first query, this is a simple English to text SQL
----> 2 db_chain.run("How many Hotels are there ?")

File /opt/conda/lib/python3.10/site-packages/langchain/chains/base.py:507, in Chain.run(self, callbacks, tags, metadata, *args, **kwargs)
    505     if len(args) != 1:
    506         raise ValueError("`run` supports only one positional argument.")
--> 507     return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[
    508         _output_key
    509     ]
    511 if kwargs and not args:
    512     return self(kwargs, callbacks=callbacks, tags=tags, metadata=metadata)[
    513         _output_key
    514     ]

File /opt/conda/lib/python3.10/site-packages/langchain/chains/base.py:312, in Chain.__call__(self, inputs, return_only_outputs, callbacks, tags, metadata, run_name, include_run_info)
    310 except BaseException as e:
    311     run_manager.on_chain_error(e)
--> 312     raise e
    313 run_manager.on_chain_end(outputs)
    314 final_outputs: Dict[str, Any] = self.prep_outputs(
    315     inputs, outputs, return_only_outputs
    316 )

File /opt/conda/lib/python3.10/site-packages/langchain/chains/base.py:306, in Chain.__call__(self, inputs, return_only_outputs, callbacks, tags, metadata, run_name, include_run_info)
    299 run_manager = callback_manager.on_chain_start(
    300     dumpd(self),
    301     inputs,
    302     name=run_name,
    303 )
    304 try:
    305     outputs = (
--> 306         self._call(inputs, run_manager=run_manager)
    307         if new_arg_supported
    308         else self._call(inputs)
    309     )
    310 except BaseException as e:
    311     run_manager.on_chain_error(e)

File /opt/conda/lib/python3.10/site-packages/langchain_experimental/sql/base.py:198, in SQLDatabaseChain._call(self, inputs, run_manager)
    194 except Exception as exc:
    195     # Append intermediate steps to exception, to aid in logging and later
    196     # improvement of few shot prompt seeds
    197     exc.intermediate_steps = intermediate_steps  # type: ignore
--> 198     raise exc

File /opt/conda/lib/python3.10/site-packages/langchain_experimental/sql/base.py:168, in SQLDatabaseChain._call(self, inputs, run_manager)
    162 _run_manager.on_text(
    163     checked_sql_command, color="green", verbose=self.verbose
    164 )
    165 intermediate_steps.append(
    166     {"sql_cmd": checked_sql_command}
    167 )  # input: sql exec
--> 168 result = self.database.run(checked_sql_command)
    169 intermediate_steps.append(str(result))  # output: sql exec
    170 sql_cmd = checked_sql_command

File /opt/conda/lib/python3.10/site-packages/langchain/utilities/sql_database.py:433, in SQLDatabase.run(self, command, fetch)
    423 def run(
    424     self,
    425     command: str,
    426     fetch: Union[Literal["all"], Literal["one"]] = "all",
    427 ) -> str:
    428     """Execute a SQL command and return a string representing the results.
    429 
    430     If the statement returns rows, a string of the results is returned.
    431     If the statement returns no rows, an empty string is returned.
    432     """
--> 433     result = self._execute(command, fetch)
    434     # Convert columns values to string to avoid issues with sqlalchemy
    435     # truncating text
    436     res = [
    437         tuple(truncate_word(c, length=self._max_string_length) for c in r.values())
    438         for r in result
    439     ]

File /opt/conda/lib/python3.10/site-packages/langchain/utilities/sql_database.py:411, in SQLDatabase._execute(self, command, fetch)
    409     else:  # postgresql and other compatible dialects
    410         connection.exec_driver_sql("SET search_path TO %s", (self._schema,))
--> 411 cursor = connection.execute(text(command))
    412 if cursor.returns_rows:
    413     if fetch == "all":

File /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1306, in Connection.execute(self, statement, *multiparams, **params)
   1302     util.raise_(
   1303         exc.ObjectNotExecutableError(statement), replace_context=err
   1304     )
   1305 else:
-> 1306     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)

File /opt/conda/lib/python3.10/site-packages/sqlalchemy/sql/elements.py:332, in ClauseElement._execute_on_connection(self, connection, multiparams, params, execution_options, _force)
    328 def _execute_on_connection(
    329     self, connection, multiparams, params, execution_options, _force=False
    330 ):
    331     if _force or self.supports_execution:
--> 332         return connection._execute_clauseelement(
    333             self, multiparams, params, execution_options
    334         )
    335     else:
    336         raise exc.ObjectNotExecutableError(self)

File /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1498, in Connection._execute_clauseelement(self, elem, multiparams, params, execution_options)
   1486 compiled_cache = execution_options.get(
   1487     "compiled_cache", self.engine._compiled_cache
   1488 )
   1490 compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
   1491     dialect=dialect,
   1492     compiled_cache=compiled_cache,
   (...)
   1496     linting=self.dialect.compiler_linting | compiler.WARN_LINTING,
   1497 )
-> 1498 ret = self._execute_context(
   1499     dialect,
   1500     dialect.execution_ctx_cls._init_compiled,
   1501     compiled_sql,
   1502     distilled_params,
   1503     execution_options,
   1504     compiled_sql,
   1505     distilled_params,
   1506     elem,
   1507     extracted_params,
   1508     cache_hit=cache_hit,
   1509 )
   1510 if has_events:
   1511     self.dispatch.after_execute(
   1512         self,
   1513         elem,
   (...)
   1517         ret,
   1518     )

File /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1862, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1859             branched.close()
   1861 except BaseException as e:
-> 1862     self._handle_dbapi_exception(
   1863         e, statement, parameters, cursor, context
   1864     )
   1866 return result

File /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/base.py:2043, in Connection._handle_dbapi_exception(self, e, statement, parameters, cursor, context)
   2041     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
   2042 elif should_wrap:
-> 2043     util.raise_(
   2044         sqlalchemy_exception, with_traceback=exc_info[2], from_=e
   2045     )
   2046 else:
   2047     util.raise_(exc_info[1], with_traceback=exc_info[2])

File /opt/conda/lib/python3.10/site-packages/sqlalchemy/util/compat.py:208, in raise_(***failed resolving arguments***)
    205     exception.__cause__ = replace_context
    207 try:
--> 208     raise exception
    209 finally:
    210     # credit to
    211     # https://cosmicpercolator.com/2016/01/13/exception-leaks-in-python-2-and-3/
    212     # as the __traceback__ object creates a cycle
    213     del exception, replace_context, from_, with_traceback

File /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1819, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1817                 break
   1818     if not evt_handled:
-> 1819         self.dialect.do_execute(
   1820             cursor, statement, parameters, context
   1821         )
   1823 if self._has_events or self.engine._has_events:
   1824     self.dispatch.after_cursor_execute(
   1825         self,
   1826         cursor,
   (...)
   1830         context.executemany,
   1831     )

File /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/default.py:732, in DefaultDialect.do_execute(self, cursor, statement, parameters, context)
    731 def do_execute(self, cursor, statement, parameters, context=None):
--> 732     cursor.execute(statement, parameters)

ProgrammingError: (hdbcli.dbapi.ProgrammingError) (257, 'sql syntax error: incorrect syntax near "I": line 1 col 1 (at pos 1)')
[SQL: I don't see any issues with the original SQL query. It looks good as written. Here is the query again:

```sql
SELECT COUNT(*) AS num_hotels
FROM hotel;
```

This counts the number of rows in the hotel table to get the number of unique hotels and returns the result in the num_hotels column alias. The query follows proper practices and does not contain any of the common mistakes listed.]
(Background on this error at: https://sqlalche.me/e/14/f405)


### Expected behavior

The select statement is produced correct, it just that the LLM perform checks on the SQL Statement and comment it before passing it to SQLAlchemy for execution. When i used Claude v2, 75% SQLDatabaseChain is successful, however 25% always gets back to the behaviour providing comments about whether the SQL Query has common mistakes.

Now with Claude v2.1, ALL of the questions are checked, and failing when passing to SQLAlchemy. It seems there is a problem with the SQLDatabaseChain, either the prompt or something else. Please help to fix this.

[Lab2.ipynb.zip](https://github.com/langchain-ai/langchain/files/13532157/Lab2.ipynb.zip)