internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: getredash/redash

Issue:
Google Spreadsheets: support for open by url
We currently use [gspread](https://github.com/burnash/gspread)'s [`open_by_key`](https://gspread.readthedocs.io/en/latest/#gspread.Client.open_by_key) method to load spreadsheets. But it also has the [`open_by_url`](https://gspread.readthedocs.io/en/latest/#gspread.Client.open_by_url) method.

Using the later we can add support for just providing the spreadsheet URL, which will be much more friendly to end users. 

To remain backward compatible we can check if the query is of format `http...` -> use the open by url method, otherwise use the open by key one.
Google Spreadsheets: support for open by url
We currently use [gspread](https://github.com/burnash/gspread)'s [`open_by_key`](https://gspread.readthedocs.io/en/latest/#gspread.Client.open_by_key) method to load spreadsheets. But it also has the [`open_by_url`](https://gspread.readthedocs.io/en/latest/#gspread.Client.open_by_url) method.

Using the later we can add support for just providing the spreadsheet URL, which will be much more friendly to end users. 

To remain backward compatible we can check if the query is of format `http...` -> use the open by url method, otherwise use the open by key one.


Discussion:

Resolving patch:
diff --git a/redash/query_runner/google_spreadsheets.py b/redash/query_runner/google_spreadsheets.py
--- a/redash/query_runner/google_spreadsheets.py
+++ b/redash/query_runner/google_spreadsheets.py
@@ -147,6 +147,7 @@ def request(self, *args, **kwargs):
 
 
 class GoogleSpreadsheet(BaseQueryRunner):
+
     @classmethod
     def annotate_query(cls):
         return False
@@ -190,13 +191,22 @@ def _get_spreadsheet_service(self):
     def test_connection(self):
         self._get_spreadsheet_service()
 
+    def is_url_key(self, key):
+        if key.startswith('https://'):
+            return True
+        return False
+
     def run_query(self, query, user):
         logger.debug("Spreadsheet is about to execute query: %s", query)
         key, worksheet_num = parse_query(query)
 
         try:
             spreadsheet_service = self._get_spreadsheet_service()
-            spreadsheet = spreadsheet_service.open_by_key(key)
+
+            if self.is_url_key(key):
+                spreadsheet = spreadsheet_service.open_by_url(key)
+            else:
+                spreadsheet = spreadsheet_service.open_by_key(key)
 
             data = parse_spreadsheet(spreadsheet, worksheet_num)
 


Test patch:
Repository: getredash/redash

Issue:
GA Data Source throws an error when no rows returned
### Issue Summary

Google Analytics Data Source throws `Error running query: 'rows'` when the query result is empty.
I have a pretty simple query with dimensions and filters, like:

```json
{
  "ids": "ga:177xxxxxx",
  "start_date": "2018-10-08",
  "end_date": "2018-10-12",
  "metrics": "ga:uniqueEvents",
  "dimensions": "ga:dimension1,ga:dimension3",
  "filters": "ga:dimension2==userrole;ga:eventCategory==eventcategory;ga:eventAction==enentaction;ga:dimension1!=demo"
}
```

Sometimes it returns empty result as there is no data. This results in error in redash.

### Steps to Reproduce

1. Create the Google Analytics Data Source
2. Make some query returning zero rows
3. Execute it in query editor

`Error running query: 'rows'` will be thrown. While this might be considered not a bug, I'd expect just an empty result with no errors.

### Technical details:

* Redash Version: 5.0.1
* Browser/OS: Chrome/macOS
* How did you install Redash: docker-compose

GA Data Source throws an error when no rows returned
### Issue Summary

Google Analytics Data Source throws `Error running query: 'rows'` when the query result is empty.
I have a pretty simple query with dimensions and filters, like:

```json
{
  "ids": "ga:177xxxxxx",
  "start_date": "2018-10-08",
  "end_date": "2018-10-12",
  "metrics": "ga:uniqueEvents",
  "dimensions": "ga:dimension1,ga:dimension3",
  "filters": "ga:dimension2==userrole;ga:eventCategory==eventcategory;ga:eventAction==enentaction;ga:dimension1!=demo"
}
```

Sometimes it returns empty result as there is no data. This results in error in redash.

### Steps to Reproduce

1. Create the Google Analytics Data Source
2. Make some query returning zero rows
3. Execute it in query editor

`Error running query: 'rows'` will be thrown. While this might be considered not a bug, I'd expect just an empty result with no errors.

### Technical details:

* Redash Version: 5.0.1
* Browser/OS: Chrome/macOS
* How did you install Redash: docker-compose



Discussion:

Resolving patch:
diff --git a/redash/query_runner/google_analytics.py b/redash/query_runner/google_analytics.py
--- a/redash/query_runner/google_analytics.py
+++ b/redash/query_runner/google_analytics.py
@@ -43,7 +43,7 @@ def parse_ga_response(response):
         })
 
     rows = []
-    for r in response['rows']:
+    for r in response.get('rows', []):
         d = {}
         for c, value in enumerate(r):
             column_name = response['columnHeaders'][c]['name']


Test patch:
Repository: getredash/redash

Issue:
URL query runner: URL base path doesn't need to be a required field



Discussion:
We would appreciate it if you could provide us with more info about this issue/pr!

Seems like a regression from #2318.
Wait, hm, there is no other field to build the URL to request though, https://github.com/getredash/redash/blob/c2429e92d2a5dd58cbe378b9cc06f3be969f747e/redash/query_runner/__init__.py#L146-L176

The field is kind of badly named, it's not really the "URL base path" but the actual "URL" that the field contains.
I had a feeling using a generic base class will come back to bite us here :)

You pass the URL you want to query as the query text. The idea of using a base path was to allow limiting the URLs the end user might use and/or save some typing if you happen to load many URLs from the same base path.

Resolving patch:
diff --git a/redash/query_runner/__init__.py b/redash/query_runner/__init__.py
--- a/redash/query_runner/__init__.py
+++ b/redash/query_runner/__init__.py
@@ -146,6 +146,7 @@ def _get_tables_stats(self, tables_dict):
 class BaseHTTPQueryRunner(BaseQueryRunner):
     response_error = "Endpoint returned unexpected status code"
     requires_authentication = False
+    requires_url = True
     url_title = 'URL base path'
     username_title = 'HTTP Basic Auth Username'
     password_title = 'HTTP Basic Auth Password'
@@ -168,9 +169,15 @@ def configuration_schema(cls):
                     'title': cls.password_title,
                 },
             },
-            'required': ['url'],
             'secret': ['password']
         }
+
+        if cls.requires_url or cls.requires_authentication:
+            schema['required'] = []
+
+        if cls.requires_url:
+            schema['required'] += ['url']
+
         if cls.requires_authentication:
             schema['required'] += ['username', 'password']
         return schema
diff --git a/redash/query_runner/url.py b/redash/query_runner/url.py
--- a/redash/query_runner/url.py
+++ b/redash/query_runner/url.py
@@ -2,6 +2,7 @@
 
 
 class Url(BaseHTTPQueryRunner):
+    requires_url = False
 
     @classmethod
     def annotate_query(cls):


Test patch:
Repository: getredash/redash

Issue:
Switch to Standard SQL for BigQuery as the default
I think it's time to switch to use Standard SQL as the default syntax for BigQuery.

Also, we should show table names in the schema browser in the Standard SQL format instead of the legacy one (`.` as separator instead of `:`).


Discussion:

Resolving patch:
diff --git a/redash/query_runner/big_query.py b/redash/query_runner/big_query.py
--- a/redash/query_runner/big_query.py
+++ b/redash/query_runner/big_query.py
@@ -110,7 +110,8 @@ def configuration_schema(cls):
                 },
                 'useStandardSql': {
                     "type": "boolean",
-                    'title': "Use Standard SQL (Beta)",
+                    'title': "Use Standard SQL",
+                    "default": True,
                 },
                 'location': {
                     "type": "string",
@@ -238,7 +239,9 @@ def _get_columns_schema(self, table_data):
         for column in table_data['schema']['fields']:
             columns.extend(self._get_columns_schema_column(column))
 
-        return {'name': table_data['id'], 'columns': columns}
+        project_id = self._get_project_id()
+        table_name = table_data['id'].replace("%s:" % project_id, "")
+        return {'name': table_name, 'columns': columns}
 
     def _get_columns_schema_column(self, column):
         columns = []
@@ -339,7 +342,8 @@ def configuration_schema(cls):
                 },
                 'useStandardSql': {
                     "type": "boolean",
-                    'title': "Use Standard SQL (Beta)",
+                    'title': "Use Standard SQL",
+                    "default": True,
                 },
                 'location': {
                     "type": "string",


Test patch:
Repository: getredash/redash

Issue:
Access current user id/details from Python data source 
First of all, thanks to the authors and contributors for a very interesting and promising tool.  
### Question:
Is it possible to access current user id/details from Python data source?
I would like to be able to control access to the viewed data at row level without a need of creating and maintaining multiple (in my case about 80) data sources and dashboards. My desired scenario:
    1. Store A logs into their account and accesses dashboard with KPI/Metrics.
    2. Dashboard calls Python data source(s).
    3. Python code loads data from a db table with all stores, identifies current user and filters out all records where store != Store A.
    4. Dashboard is dynamically loaded with entries for Store A only.

Also if there are any other ways to achieve the above, please advise.   

### Technical details:
* Redash Version: 0.12.0


Discussion:
Also interested in something along these lines!

Resolving patch:
diff --git a/redash/query_runner/python.py b/redash/query_runner/python.py
--- a/redash/query_runner/python.py
+++ b/redash/query_runner/python.py
@@ -211,10 +211,15 @@ def get_query_result(query_id):
 
         return json_loads(query.latest_query_data.data)
 
+    def get_current_user(self):
+        return self._current_user.to_dict()
+
     def test_connection(self):
         pass
 
     def run_query(self, query, user):
+        self._current_user = user
+
         try:
             error = None
 
@@ -239,6 +244,7 @@ def run_query(self, query, user):
             restricted_globals = dict(__builtins__=builtins)
             restricted_globals["get_query_result"] = self.get_query_result
             restricted_globals["get_source_schema"] = self.get_source_schema
+            restricted_globals["get_current_user"] = self.get_current_user
             restricted_globals["execute_query"] = self.execute_query
             restricted_globals["add_result_column"] = self.add_result_column
             restricted_globals["add_result_row"] = self.add_result_row


Test patch:
Repository: getredash/redash

Issue:
Exporting to Excel file fails when one of the columns is a dictionary
<!--

We use GitHub only for bug reports 🐛

Anything else should be posted to https://discuss.redash.io 👫

🚨For support, help & questions use https://discuss.redash.io/c/support
💡For feature requests & ideas use https://discuss.redash.io/c/feature-requests

**Found a security vulnerability?** Please email security@redash.io to report any security vulnerabilities. We will acknowledge receipt of your vulnerability and strive to send you regular updates about our progress. If you're curious about the status of your disclosure please feel free to email us again. If you want to encrypt your disclosure email, you can use this PGP key.

-->

### Issue Summary

get error when exporting query results to excel file:
![image](https://user-images.githubusercontent.com/33534430/51517880-30217d80-1e57-11e9-9cbf-0ef7ad187218.png)
environment:ec2 on ecs


### Steps to Reproduce

1. create a new query
2. excute the query,save it ,then download it as excel file

Any other info e.g. Why do you consider this to be a bug? What did you expect to happen instead?

### Technical details:

* Redash Version:6.0.0+b8537
* Browser/OS:chrome
* How did you install Redash:run redash by  ecs  on aws 



Discussion:
Can you show the error log more detail? And can you download CSV successfully?
> Can you show the error log more detail? And can you download CSV successfully?

![image](https://user-images.githubusercontent.com/33534430/51521827-b7282300-1e62-11e9-9375-d5a119fd29ab.png)
i can download csv successfully,but get error when downloading excel ,and i found that there is a field whose attribute is json.that maybe the cause,because I can export query without json attribute to excel successfully.
BTW,the data source is postgres.
> Can you show the error log more detail? And can you download CSV successfully?

I see the similiar problem has benn sloved in issue #2060,is this the same problem?

Indeed it looks like a similar issue.

Resolving patch:
diff --git a/redash/models/__init__.py b/redash/models/__init__.py
--- a/redash/models/__init__.py
+++ b/redash/models/__init__.py
@@ -351,7 +351,7 @@ def make_excel_content(self):
         for (r, row) in enumerate(query_data['rows']):
             for (c, name) in enumerate(column_names):
                 v = row.get(name)
-                if isinstance(v, list):
+                if isinstance(v, list) or isinstance(v, dict):
                     v = str(v).encode('utf-8')
                 sheet.write(r + 1, c, v)
 


Test patch:
Repository: getredash/redash

Issue:
Clickhouse: password is optional but we try to access it anyway
For Clickhouse type data sources, we don't require a password. But the code does require it by trying to directly access the value in the options dictionary, instead of using `get`:

https://github.com/getredash/redash/blob/823e4ccdd6fcfee5d0df0d919d87af3100876549/redash/query_runner/clickhouse.py#L77


Discussion:

Resolving patch:
diff --git a/redash/query_runner/clickhouse.py b/redash/query_runner/clickhouse.py
--- a/redash/query_runner/clickhouse.py
+++ b/redash/query_runner/clickhouse.py
@@ -68,13 +68,13 @@ def _get_tables(self, schema):
 
     def _send_query(self, data, stream=False):
         r = requests.post(
-            self.configuration['url'],
+            self.configuration.get('url', "http://127.0.0.1:8123"),
             data=data.encode("utf-8"),
             stream=stream,
             timeout=self.configuration.get('timeout', 30),
             params={
-                'user': self.configuration['user'],
-                'password':  self.configuration['password'],
+                'user': self.configuration.get('user', "default"),
+                'password':  self.configuration.get('password', ""),
                 'database': self.configuration['dbname']
             }
         )


Test patch:
Repository: getredash/redash

Issue:
When creating a visualisation after running a query the query results are deleted
We are testing the latest stable version of redash (6.0.0+b10818 (7fa66654)
Found a regression when creating a new visualization, to reproduce:
0. Open a new browser tab and log into redash
1. Run a query
2. Create a new visualization
3. The X & Y drop downs are empty
4. When you cancel the visualization creation you see that the query results disappeared

Note - the issue occurs only when you run redash for the first time on a new session


Discussion:
The issue isn't with a specific visualization, but with the query losing reference to the query result.

I think the issue happens because we don't take reference to the query result from the UI, so when first saving the query it gets saved without a query result reference (`latest_query_data_id`):

https://github.com/getredash/redash/blob/933dd753a8197708fe37432ee037a7f6bba258d5/redash/handlers/queries.py#L213-L214



Resolving patch:
diff --git a/redash/handlers/queries.py b/redash/handlers/queries.py
--- a/redash/handlers/queries.py
+++ b/redash/handlers/queries.py
@@ -228,7 +228,7 @@ def post(self):
             'object_type': 'query'
         })
 
-        return QuerySerializer(query).serialize()
+        return QuerySerializer(query, with_visualizations=True).serialize()
 
 
 class QueryArchiveResource(BaseQueryListResource):


Test patch:
Repository: getredash/redash

Issue:
Widget turns into a textbox when a visualization is removed
Current behavior: when deleting a visualization in use in some dashboard, the widget using it turns into a text box.

Expected behavior: the widget should be removed.

Bonus behavior: warn the user if the visualization is currently in use before deleting.


Discussion:
@arikfr Are you sure that this is frontend-related bug? I think the backend should search for dashboards that use the visualizations and update them. On a frontend we can show a placeholder instead of vis ("Visualization is not available" or something like that) - it's the most we can do with JS.
It's indeed a backend issue, I miscategorized.

Resolving patch:
diff --git a/redash/models/__init__.py b/redash/models/__init__.py
--- a/redash/models/__init__.py
+++ b/redash/models/__init__.py
@@ -922,7 +922,7 @@ def copy(self):
 class Widget(TimestampMixin, BelongsToOrgMixin, db.Model):
     id = Column(db.Integer, primary_key=True)
     visualization_id = Column(db.Integer, db.ForeignKey('visualizations.id'), nullable=True)
-    visualization = db.relationship(Visualization, backref='widgets')
+    visualization = db.relationship(Visualization, backref=backref('widgets', cascade='delete'))
     text = Column(db.Text, nullable=True)
     width = Column(db.Integer)
     options = Column(db.Text)


Test patch:
diff --git a/tests/handlers/test_visualizations.py b/tests/handlers/test_visualizations.py
--- a/tests/handlers/test_visualizations.py
+++ b/tests/handlers/test_visualizations.py
@@ -128,3 +128,11 @@ def test_only_owner_collaborator_or_admin_can_delete_visualization(self):
 
         rv = self.make_request('delete', path, user=admin_from_diff_org)
         self.assertEquals(rv.status_code, 404)
+
+    def test_deleting_a_visualization_deletes_dashboard_widgets(self):
+        vis = self.factory.create_visualization()
+        widget = self.factory.create_widget(visualization=vis)
+
+        rv = self.make_request('delete', '/api/visualizations/{}'.format(vis.id))
+
+        self.assertIsNone(models.Widget.query.filter(models.Widget.id == widget.id).first())
Repository: getredash/redash

Issue:
PostgreSQL queries returning a tsrange fail
### Issue Summary

Running a query which includes a tsrange column fail with a JSON error like this:

```
DateTimeRange(datetime.datetime(2016, 8, 25, 15, 21, 45, 760521), datetime.datetime(2016, 10, 14, 5, 56, 52, 452297), '[)') is not JSON serializable
```

### Steps to Reproduce

1. Create a table with a tsrange column
2. Insert some data into it
3. Create a `SELECT * FROM table` query
4. Run the query

This is turning out to be confusing for our users - I'm getting regular complaints about queries failing with this error because some of our tables have range columns.

### Technical details:

* Redash Version: 1.0.1+b2845
* Browser/OS: Linux
* How did you install Redash: Docker image



Discussion:
We need to implement a custom JSON encoder, as we did for MongoDB and other data sources, that will properly handle this class type.

Resolving patch:
diff --git a/redash/query_runner/pg.py b/redash/query_runner/pg.py
--- a/redash/query_runner/pg.py
+++ b/redash/query_runner/pg.py
@@ -3,9 +3,10 @@
 import select
 
 import psycopg2
+from psycopg2.extras import Range
 
 from redash.query_runner import *
-from redash.utils import json_dumps, json_loads
+from redash.utils import JSONEncoder, json_dumps, json_loads
 
 logger = logging.getLogger(__name__)
 
@@ -28,6 +29,26 @@
 }
 
 
+class PostgreSQLJSONEncoder(JSONEncoder):
+    def default(self, o):
+        if isinstance(o, Range):
+            # From: https://github.com/psycopg/psycopg2/pull/779
+            if o._bounds is None:
+                return ''
+
+            items = [
+                o._bounds[0],
+                str(o._lower),
+                ', ',
+                str(o._upper),
+                o._bounds[1]
+            ]
+
+            return ''.join(items)
+
+        return super(PostgreSQLJSONEncoder, self).default(o)
+
+
 def _wait(conn, timeout=None):
     while 1:
         try:
@@ -165,7 +186,7 @@ def run_query(self, query, user):
 
                 data = {'columns': columns, 'rows': rows}
                 error = None
-                json_data = json_dumps(data, ignore_nan=True)
+                json_data = json_dumps(data, ignore_nan=True, cls=PostgreSQLJSONEncoder)
             else:
                 error = 'Query completed but it returned no data.'
                 json_data = None


Test patch:
Repository: getredash/redash

Issue:
Verify address when users change their e-mail
### Issue Summary

Currently, e-mail address are verified automatically by the fact that users receive an e-mail invitation and accept it. However, users can later on change their e-mail address and no verification is currently done in that scenario.


Discussion:

Resolving patch:
diff --git a/redash/handlers/users.py b/redash/handlers/users.py
--- a/redash/handlers/users.py
+++ b/redash/handlers/users.py
@@ -13,7 +13,7 @@
     require_permission_or_owner, require_admin
 from redash.handlers.base import BaseResource, require_fields, get_object_or_404, paginate, order_results as _order_results
 
-from redash.authentication.account import invite_link_for_user, send_invite_email, send_password_reset_email
+from redash.authentication.account import invite_link_for_user, send_invite_email, send_password_reset_email, send_verify_email
 from redash.settings import parse_boolean
 
 
@@ -225,10 +225,17 @@ def post(self, user_id):
             if domain.lower() in blacklist or domain.lower() == 'qq.com':
                 abort(400, message='Bad email address.')
 
+        email_changed = 'email' in params and params['email'] != user.email
+        if email_changed:
+            user.is_email_verified = False
+
         try:
             self.update_model(user, params)
             models.db.session.commit()
 
+            if email_changed:
+                send_verify_email(user, self.current_org)
+
             # The user has updated their email or password. This should invalidate all _other_ sessions,
             # forcing them to log in again. Since we don't want to force _this_ session to have to go
             # through login again, we call `login_user` in order to update the session with the new identity details.


Test patch:
diff --git a/tests/handlers/test_users.py b/tests/handlers/test_users.py
--- a/tests/handlers/test_users.py
+++ b/tests/handlers/test_users.py
@@ -202,6 +202,12 @@ def test_returns_200_for_non_admin_changing_his_own(self):
         rv = self.make_request('post', "/api/users/{}".format(self.factory.user.id), data={"name": "New Name"})
         self.assertEqual(rv.status_code, 200)
 
+    def test_marks_email_as_not_verified_when_changed(self):
+        user = self.factory.user
+        user.is_email_verified = True
+        rv = self.make_request('post', "/api/users/{}".format(user.id), data={"email": "donald@trump.biz"})
+        self.assertFalse(user.is_email_verified)
+
     def test_returns_200_for_admin_changing_other_user(self):
         admin = self.factory.create_admin()