internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: getredash/redash

Issue:
Ability to specify recipient email address when using send_test_email
It would be useful if manage.py send_test_email allowed you to specify a recipient instead of sending the test to whatever address is specified in REDASH_MAIL_DEFAULT_SENDER="".

I run re:dash on a subdomain and am using Mailgun for alerts, but since I use Google Apps and their MX records for my domain, my Mailgun address can't deliver email to itself.



Discussion:

Resolving patch:
diff --git a/manage.py b/manage.py
--- a/manage.py
+++ b/manage.py
@@ -47,12 +47,16 @@ def check_settings():
     for name, item in settings.all_settings().iteritems():
         print "{} = {}".format(name, item)
 
-@manager.command
-def send_test_mail():
+
+@manager.option('email', default=None, help="Email address to send test message to (default: the address you defined in MAIL_DEFAULT_SENDER)")
+def send_test_mail(email=None):
     from redash import mail
     from flask_mail import Message
 
-    mail.send(Message(subject="Test Message from re:dash", recipients=[settings.MAIL_DEFAULT_SENDER], body="Test message."))
+    if email is None:
+        email = settings.MAIL_DEFAULT_SENDER
+
+    mail.send(Message(subject="Test Message from re:dash", recipients=[email], body="Test message."))
 
 
 if __name__ == '__main__':


Test patch:
Repository: getredash/redash

Issue:
Wrong sort for Date column
It happens if sort by date column with UI.

Screenshot shows everything. 

<img width="544" alt="2016-12-06 14 37 07" src="https://cloud.githubusercontent.com/assets/7091907/20924299/dea511b4-bbc1-11e6-8ffd-2bdda4bcbbd8.png">



Discussion:
Sorry, didn't get the problem. 

It seems to be a descending sort. 
**2016-12-06** is greater than **2016-11-07**, right?
@WesleyBatista but it's ascending sort. Looks like it sorts strings 'DD/MM/YYYY' but not dates 'YYYY-MM-DD'

Resolving patch:
diff --git a/redash/handlers/queries.py b/redash/handlers/queries.py
--- a/redash/handlers/queries.py
+++ b/redash/handlers/queries.py
@@ -58,10 +58,6 @@ def post(self):
         for field in ['id', 'created_at', 'api_key', 'visualizations', 'latest_query_data', 'last_modified_by']:
             query_def.pop(field, None)
 
-        # If we already executed this query, save the query result reference
-        if 'latest_query_data_id' in query_def:
-            query_def['latest_query_data'] = query_def.pop('latest_query_data_id')
-
         query_def['query_text'] = query_def.pop('query')
         query_def['user'] = self.current_user
         query_def['data_source'] = data_source


Test patch:
Repository: getredash/redash

Issue:
Elasticsearch query runner: mappings_data is not defined
### Issue Summary
When executing an elasticsearch query, a message appears that "mappings_data is not defined".

### Steps to Reproduce
1. Clone the latest version (master branch) from github and install the dev environment.
2. Add an elasticsearch instance and try to execute a query, e.g.
`{"query": {'"match_all": {}}`
3. A message appears that the query was not successfull (mappings_data is not defined)

The bug is quite obvious since https://github.com/getredash/redash/blob/3bc98fba5bd2b6b723ba0d0ce3007a651e8c2acb/redash/query_runner/elasticsearch.py#L126 refers to mappings_data, which is nowhere defined in the same file. Unfortunately I didn't find out what this mappings_data should contain.

### Technical details:

* Redash Version: master branch
* Browser/OS: Linux/Chrome




Discussion:

Resolving patch:
diff --git a/redash/query_runner/elasticsearch.py b/redash/query_runner/elasticsearch.py
--- a/redash/query_runner/elasticsearch.py
+++ b/redash/query_runner/elasticsearch.py
@@ -123,8 +123,8 @@ def _get_query_mappings(self, url):
         if error:
             return mappings, error
 
-        for index_name in mappings_data:
-            index_mappings = mappings_data[index_name]
+        for index_name in mappings:
+            index_mappings = mappings[index_name]
             for m in index_mappings.get("mappings", {}):
                 for property_name in index_mappings["mappings"][m]["properties"]:
                     property_data = index_mappings["mappings"][m]["properties"][property_name]


Test patch:
Repository: getredash/redash

Issue:
mssql get_schema() fails on case sensitive object name setups
### Issue Summary

get_schema() for MS-SQL fails on case sensitive object name setups.

### Steps to Reproduce

1. Define datasource for  case-sensitive-object-name MSSQL database instance.

### Technical details:

* Redash Version: 0.12.0

### How to fix

change
`        SELECT table_schema, table_name, column_name
        FROM information_schema.columns
        WHERE ...`
to
`        SELECT table_schema, table_name, column_name
        FROM INFORMATION_SCHEMA.COLUMNS
        WHERE...`
fixed my case.

### Where to go from here
I'm not really sure if this is the best way to fix this issue. If any MSSQL instance can be case-sensitive AND has this tablename in lower case, then this fix will not work. But I don't know if this can happen.


Discussion:

Resolving patch:
diff --git a/redash/query_runner/mssql.py b/redash/query_runner/mssql.py
--- a/redash/query_runner/mssql.py
+++ b/redash/query_runner/mssql.py
@@ -94,7 +94,7 @@ def __init__(self, configuration):
     def _get_tables(self, schema):
         query = """
         SELECT table_schema, table_name, column_name
-        FROM information_schema.columns
+        FROM INFORMATION_SCHEMA.COLUMNS
         WHERE table_schema NOT IN ('guest','INFORMATION_SCHEMA','sys','db_owner','db_accessadmin'
                                   ,'db_securityadmin','db_ddladmin','db_backupoperator','db_datareader'
                                   ,'db_datawriter','db_denydatareader','db_denydatawriter'


Test patch:
Repository: getredash/redash

Issue:
Cannot download CSV from MongoDB Results when rows have missing columns
This works fine in the Excel download however. 
  
```
File "/redash/handlers/query_results.py", line 172, in get
    response = self.make_csv_response(query_result)
  File "/redash/handlers/query_results.py", line 199, in make_csv_response
    writer.writerow(row)
  File "/usr/lib64/python2.7/csv.py", line 148, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
  File "/usr/lib64/python2.7/csv.py", line 144, in _dict_to_list
    ", ".join(wrong_fields))
ValueError: dict contains fields not in fieldnames: _id
```
Cannot download CSV from MongoDB Results when rows have missing columns
This works fine in the Excel download however. 
  
```
File "/redash/handlers/query_results.py", line 172, in get
    response = self.make_csv_response(query_result)
  File "/redash/handlers/query_results.py", line 199, in make_csv_response
    writer.writerow(row)
  File "/usr/lib64/python2.7/csv.py", line 148, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
  File "/usr/lib64/python2.7/csv.py", line 144, in _dict_to_list
    ", ".join(wrong_fields))
ValueError: dict contains fields not in fieldnames: _id
```


Discussion:
The Excel serializer knows to handle missing columns, while the CSV one uses a dict writer which expects all the fields to be present in every row.
(btw, it's not `_id` specific)
The Excel serializer knows to handle missing columns, while the CSV one uses a dict writer which expects all the fields to be present in every row.
(btw, it's not `_id` specific)

Resolving patch:
diff --git a/redash/models.py b/redash/models.py
--- a/redash/models.py
+++ b/redash/models.py
@@ -653,7 +653,7 @@ def make_csv_content(self):
         s = cStringIO.StringIO()
 
         query_data = json.loads(self.data)
-        writer = csv.DictWriter(s, fieldnames=[col['name'] for col in query_data['columns']])
+        writer = csv.DictWriter(s, extrasaction="ignore", fieldnames=[col['name'] for col in query_data['columns']])
         writer.writer = utils.UnicodeWriter(s)
         writer.writeheader()
         for row in query_data['rows']:


Test patch:
Repository: getredash/redash

Issue:
Make dashboard level filters a feature available to non-admins
### Issue Summary

Currently to enable dashboard level filters you have to be an administrator and you have to change a flag manually in the dashboards table. It would be good if this was just on by default or an option that users could change through the front end.

### Technical details:

* Redash Version: 1.0.3
* Browser/OS: Chrome
* How did you install Redash: Amazon via the AMI



Discussion:

Resolving patch:
diff --git a/redash/handlers/dashboards.py b/redash/handlers/dashboards.py
--- a/redash/handlers/dashboards.py
+++ b/redash/handlers/dashboards.py
@@ -129,7 +129,7 @@ def post(self, dashboard_slug):
         require_object_modify_permission(dashboard, self.current_user)
 
         updates = project(dashboard_properties, ('name', 'layout', 'version',
-                                                 'is_draft'))
+                                                 'is_draft', 'dashboard_filters_enabled'))
 
         # SQLAlchemy handles the case where a concurrent transaction beats us
         # to the update. But we still have to make sure that we're not starting


Test patch:
Repository: getredash/redash

Issue:
Log lines go to `stderr` and not to `stdout`
### Issue Summary

Running redash (dockerized) is logging everything to `stderr`. I think this is the default for python's `logging` module, although I'm not entirely sure. For me it makes sense that `INFO` level log lines should go into `stdout`, so having some kind of option to configure that would be nice.

### Steps to Reproduce

1. run redash in docker and redirect both `stderr` and `stdout` to separate files:
`docker run --rm redash/redash server 1> ~/out.log 2> ~/error.log`

2. `out.log` file remains empty whereas all the log lines go to `error.log`.

### Technical details:

* Redash Version: 1.0.3
* Browser/OS: -
* How did you install Redash: docker



Discussion:

Resolving patch:
diff --git a/redash/__init__.py b/redash/__init__.py
--- a/redash/__init__.py
+++ b/redash/__init__.py
@@ -1,4 +1,5 @@
 import os
+import sys
 import logging
 import urlparse
 import redis
@@ -21,7 +22,7 @@
 
 
 def setup_logging():
-    handler = logging.StreamHandler()
+    handler = logging.StreamHandler(sys.stdout if settings.LOG_STDOUT else sys.stderr)
     formatter = logging.Formatter('[%(asctime)s][PID:%(process)d][%(levelname)s][%(name)s] %(message)s')
     handler.setFormatter(formatter)
     logging.getLogger().addHandler(handler)
diff --git a/redash/settings.py b/redash/settings.py
--- a/redash/settings.py
+++ b/redash/settings.py
@@ -134,6 +134,7 @@ def all_settings():
 SESSION_COOKIE_SECURE = parse_boolean(os.environ.get("REDASH_SESSION_COOKIE_SECURE") or str(ENFORCE_HTTPS))
 
 LOG_LEVEL = os.environ.get("REDASH_LOG_LEVEL", "INFO")
+LOG_STDOUT = parse_boolean(os.environ.get('REDASH_LOG_STDOUT', 'false'))
 
 # Mail settings:
 MAIL_SERVER = os.environ.get('REDASH_MAIL_SERVER', 'localhost')


Test patch:
Repository: getredash/redash

Issue:
Redshift: Support ACM Certificates for SSL Connections
https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-transitioning-to-acm-certs.html


Discussion:

Resolving patch:
diff --git a/redash/query_runner/pg.py b/redash/query_runner/pg.py
--- a/redash/query_runner/pg.py
+++ b/redash/query_runner/pg.py
@@ -1,3 +1,4 @@
+import os
 import json
 import logging
 import select
@@ -130,14 +131,19 @@ def _get_tables(self, schema):
 
         return schema.values()
 
-    def run_query(self, query, user):
+    def _get_connection(self):
         connection = psycopg2.connect(user=self.configuration.get('user'),
                                       password=self.configuration.get('password'),
                                       host=self.configuration.get('host'),
                                       port=self.configuration.get('port'),
                                       dbname=self.configuration.get('dbname'),
+                                      sslmode=self.configuration.get('sslmode'),
                                       async=True)
 
+        return connection
+
+    def run_query(self, query, user):
+        connection = self._get_connection()
         _wait(connection, timeout=10)
 
         cursor = connection.cursor()
@@ -177,8 +183,23 @@ class Redshift(PostgreSQL):
     def type(cls):
         return "redshift"
 
+    def _get_connection(self):
+        sslrootcert_path = os.path.join(os.path.dirname(__file__), './files/redshift-ca-bundle.crt')
+
+        connection = psycopg2.connect(user=self.configuration.get('user'),
+                                      password=self.configuration.get('password'),
+                                      host=self.configuration.get('host'),
+                                      port=self.configuration.get('port'),
+                                      dbname=self.configuration.get('dbname'),
+                                      sslmode='require',
+                                      sslrootcert=sslrootcert_path,
+                                      async=True)
+
+        return connection
+
     @classmethod
     def configuration_schema(cls):
+
         return {
             "type": "object",
             "properties": {
@@ -199,6 +220,7 @@ def configuration_schema(cls):
                     "title": "Database Name"
                 }
             },
+            "order": ['host', 'port', 'user', 'password'],
             "required": ["dbname", "user", "password", "host", "port"],
             "secret": ["password"]
         }


Test patch:
Repository: getredash/redash

Issue:
Parameters are loosing their configuration on fork
### Issue Summary
When forking a query, its parameters are loosing their configuration.

### Steps to Reproduce

1. Create query with parameters
2. Fork it

Any other info e.g. Why do you consider this to be a bug? What did you expect to happen instead?

### Technical details:

* Redash Version: commit 6982f48b34a66ef1aeb9fe42065e2e2a412e0d54
* How did you install Redash: docker



Discussion:

Resolving patch:
diff --git a/redash/models.py b/redash/models.py
--- a/redash/models.py
+++ b/redash/models.py
@@ -967,7 +967,7 @@ def get_by_id(cls, _id):
 
     def fork(self, user):
         forked_list = ['org', 'data_source', 'latest_query_data', 'description',
-                       'query_text', 'query_hash']
+                       'query_text', 'query_hash', 'options']
         kwargs = {a: getattr(self, a) for a in forked_list}
         forked_query = Query.create(name=u'Copy of (#{}) {}'.format(self.id, self.name),
                                     user=user, **kwargs)


Test patch:
Repository: getredash/redash

Issue:
Use can't download dataset before saving query
Because the query results url contains the query id, before saving the user can't download the dataset. 

We need to allow addressing query results without query id.



Discussion:
Fixed.

I am facing this issue in latest version

Resolving patch:
diff --git a/redash/handlers/api.py b/redash/handlers/api.py
--- a/redash/handlers/api.py
+++ b/redash/handlers/api.py
@@ -79,6 +79,7 @@ def json_representation(data, code, headers=None):
 
 api.add_org_resource(QueryResultListResource, '/api/query_results', endpoint='query_results')
 api.add_org_resource(QueryResultResource,
+                     '/api/query_results/<query_result_id>.<filetype>',
                      '/api/query_results/<query_result_id>',
                      '/api/queries/<query_id>/results.<filetype>',
                      '/api/queries/<query_id>/results/<query_result_id>.<filetype>',


Test patch:
Repository: getredash/redash

Issue:
ElasticSearch query fails on ES 6
### Issue Summary

Using the public docker images which seem to be based on Redash 4 (beta?) trying to setup a query for an ES 6 backend results in

    Error running query: Failed to execute query. Return Code: 500 Reason: {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"source and source_content_type parameters are required"}],"type":"illegal_state_exception","reason":"source and source_content_type parameters are required"},"status":500}

### Steps to Reproduce

1. Deploy redash using the redash docker compose file
2. Deploy ES 6 backend
3. Setup ElasticSearch datasource in Redash
4. Try a simple query

### Technical details:

* Redash Version: 4.0.0 (not sure about the exact built number - can't determine it from the docker image)
* Browser/OS: Redash in docker on Ubuntu 17.10
* How did you install Redash: redash docker compose file from github pulling in redash:latest from docker hub

Looks like redash uses the `source` query parameter but as per https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html this also requires to provide a `source_content_type` parameter.



Discussion:
Thanks @chriswue. This sounds like a support issue; have you opened a thread on the [forum](https://discuss.redash.io/)? 
@RichardLitt No, I assumed this was a bug in redash. I will open a support thread
Opened support thread: https://discuss.redash.io/t/connecting-redash-4-to-elasticsearch-6/1412
@RichardLitt Is this really a support issue?

If I go to the [relevant line](https://github.com/getredash/redash/blob/cf436e45a41d149a8133eca8bb635496e4020df2/redash/query_runner/elasticsearch.py#L423) and add the appropriate content type for my query, i.e. change it to;

`params = {"source": json.dumps(query_dict), "source_content_type": "application/json"}`

then my queries execute as expected.

@chriswue Did you get around this?
@alexdrans Good point. No, I guess it is not. Thanks for the work, there. I'll flag this as a bug. 
@alexdrans I haven't found a work-around, needs fixing in Redash. Would have submitted a PR but got side tracked by something else.

Resolving patch:
diff --git a/redash/query_runner/elasticsearch.py b/redash/query_runner/elasticsearch.py
--- a/redash/query_runner/elasticsearch.py
+++ b/redash/query_runner/elasticsearch.py
@@ -420,7 +420,7 @@ def run_query(self, query, user):
             if error:
                 return None, error
 
-            params = {"source": json.dumps(query_dict)}
+            params = {"source": json.dumps(query_dict), "source_content_type": "application/json"}
             logger.debug("Using URL: %s", url)
             logger.debug("Using params : %s", params)
             r = requests.get(url, params=params, auth=self.auth)


Test patch:
Repository: getredash/redash

Issue:
Properly show RECORD fields in BigQuery's schema
This RECORD:

![image](https://cloud.githubusercontent.com/assets/71468/20478473/ebeb428e-afe2-11e6-8dde-531415041eda.png)

Will be shown in Redash just as `location_data`. Instead we should deconstruct it into `location_data.city_name`, `location_data.country_name`, etc.


Discussion:

Resolving patch:
diff --git a/redash/query_runner/big_query.py b/redash/query_runner/big_query.py
--- a/redash/query_runner/big_query.py
+++ b/redash/query_runner/big_query.py
@@ -205,7 +205,8 @@ def _get_query_result(self, jobs, query):
 
         data = {
             "columns": columns,
-            "rows": rows
+            "rows": rows, 
+            'metadata': {'data_scanned': query_reply['totalBytesProcessed']}
         }
 
         return data
@@ -224,7 +225,14 @@ def get_schema(self, get_stats=False):
             for table in tables.get('tables', []):
                 table_data = service.tables().get(projectId=project_id, datasetId=dataset_id, tableId=table['tableReference']['tableId']).execute()
 
-                schema.append({'name': table_data['id'], 'columns': map(lambda r: r['name'], table_data['schema']['fields'])})
+                columns = []
+                for column in table_data['schema']['fields']:
+                    if column['type'] == 'RECORD':
+                        for field in column['fields']:
+                            columns.append(u"{}.{}".format(column['name'], field['name']))
+                    else:
+                        columns.append(column['name'])
+                schema.append({'name': table_data['id'], 'columns': columns})
 
         return schema
 


Test patch:
Repository: getredash/redash

Issue:
Non blocking widget refresh indicator
When refreshing a dashboard widget the previous results are hidden by the refresh animation. This can be an issue when refreshing a dashboard frequently, as you might happen to see the spinner for long period of times.

To solve this we can keep showing the old data until new one is available, while showing some indication that refresh is in progress.

Is the following animation enough?

![](http://g.recordit.co/CyccMD6dFc.gif)
After refreshing a dashboard, widgets become draggable even when not in edit mode



Discussion:
@kocsmy 
Yes @arikfr I think this is great and that animation should be enough. 

I assume the frame rate is low because of the recording/gif and it should be smooth movement.


Resolving patch:
diff --git a/redash/handlers/widgets.py b/redash/handlers/widgets.py
--- a/redash/handlers/widgets.py
+++ b/redash/handlers/widgets.py
@@ -44,7 +44,7 @@ def post(self):
         models.db.session.commit()
 
         models.db.session.commit()
-        return {'widget': widget.to_dict()}
+        return widget.to_dict()
 
 
 class WidgetResource(BaseResource):


Test patch:
diff --git a/tests/handlers/test_widgets.py b/tests/handlers/test_widgets.py
--- a/tests/handlers/test_widgets.py
+++ b/tests/handlers/test_widgets.py
@@ -54,7 +54,7 @@ def test_create_text_widget(self):
         rv = self.make_request('post', '/api/widgets', data=data)
 
         self.assertEquals(rv.status_code, 200)
-        self.assertEquals(rv.json['widget']['text'], 'Sample text.')
+        self.assertEquals(rv.json['text'], 'Sample text.')
 
     def test_delete_widget(self):
         widget = self.factory.create_widget()
Repository: getredash/redash

Issue:
elastic search datasource query error
redash version 4.1
any query will return error:

Error running query: Failed to execute query. Return Code: 400 Reason: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/app_api_log_2018-07-20/_search] contains unrecognized parameter: [source_content_type]"}],"type":"illegal_argument_exception","reason":"request [/app_api_log_2018-07-20/_search] contains unrecognized parameter: [source_content_type]"},"status":400}

but i never use "source_content_type" in my query.is redash pass this unrecognized param to es? will u guys fix this issue?


Discussion:
Hi @jesusslim. I think #2267 changes this behavior.

Resolving patch:
diff --git a/redash/query_runner/elasticsearch.py b/redash/query_runner/elasticsearch.py
--- a/redash/query_runner/elasticsearch.py
+++ b/redash/query_runner/elasticsearch.py
@@ -420,10 +420,9 @@ def run_query(self, query, user):
             if error:
                 return None, error
 
-            params = {"source": json.dumps(query_dict), "source_content_type": "application/json"}
             logger.debug("Using URL: %s", url)
-            logger.debug("Using params : %s", params)
-            r = requests.get(url, params=params, auth=self.auth)
+            logger.debug("Using query: %s", query_dict)
+            r = requests.get(url, json=query_dict, auth=self.auth)
             r.raise_for_status()
             logger.debug("Result: %s", r.json())
 


Test patch: