Repository: getredash/redash

Issue:
JQL: add support for fetching all the results by way of pagination
### Issue Summary

The JQL integration returns only the first 50 issues. This is the default number of issues returned via JIRA REST API. A mechanism should be implemented where a query is executed multiple times to fetch subsequent issues from JIRA.

### Steps to Reproduce

1. Configure JIRA integration
2. Create any JQL query which returns more than 50 issues
3. Execute the query

Expected: More than 50 issues returned.
Actual: Only 50 issues returned.

### Technical details:

* Redash Version: 0.12.0+b2449
* Browser/OS: Ubuntu 16.4
* How did you install Redash: Provisioning script from https://redash.io/help-onpremise/setup/setting-up-redash-instance.html
JQL: add support for fetching all the results by way of pagination
### Issue Summary

The JQL integration returns only the first 50 issues. This is the default number of issues returned via JIRA REST API. A mechanism should be implemented where a query is executed multiple times to fetch subsequent issues from JIRA.

### Steps to Reproduce

1. Configure JIRA integration
2. Create any JQL query which returns more than 50 issues
3. Execute the query

Expected: More than 50 issues returned.
Actual: Only 50 issues returned.

### Technical details:

* Redash Version: 0.12.0+b2449
* Browser/OS: Ubuntu 16.4
* How did you install Redash: Provisioning script from https://redash.io/help-onpremise/setup/setting-up-redash-instance.html


Discussion:
You can set `"maxResults"` option to get more than 50 rows. It is not a bug, it's default JQL limitation.
Merged #1721.

While @denisov-vlad is right that you can just specify this in a query, it looks like 1K is a reasonable limit.
It doesn't look like the 1K bump resolves this issue. The API [limits the results to 100](https://jira.atlassian.com/browse/JRACLOUD-67570?_ga=2.214036247.1681779146.1547612561-1966547513.1547612561) so setting `maxResults` above that is effectively ignored. To return more than 100 results, we would have to [paginate ](https://developer.atlassian.com/cloud/jira/platform/rest/v3/?_ga=2.15756085.2024100639.1505889530-892835306.1502362635#pagination) through the API. 

Could we re-open or should I create another issue to implement pagination?
@justmiles I reopened this.
You can set `"maxResults"` option to get more than 50 rows. It is not a bug, it's default JQL limitation.
Merged #1721.

While @denisov-vlad is right that you can just specify this in a query, it looks like 1K is a reasonable limit.
It doesn't look like the 1K bump resolves this issue. The API [limits the results to 100](https://jira.atlassian.com/browse/JRACLOUD-67570?_ga=2.214036247.1681779146.1547612561-1966547513.1547612561) so setting `maxResults` above that is effectively ignored. To return more than 100 results, we would have to [paginate ](https://developer.atlassian.com/cloud/jira/platform/rest/v3/?_ga=2.15756085.2024100639.1505889530-892835306.1502362635#pagination) through the API. 

Could we re-open or should I create another issue to implement pagination?
@justmiles I reopened this.

Resolving patch:
diff --git a/redash/query_runner/jql.py b/redash/query_runner/jql.py
--- a/redash/query_runner/jql.py
+++ b/redash/query_runner/jql.py
@@ -24,6 +24,8 @@ def add_column(self, column, column_type=TYPE_STRING):
     def to_json(self):
         return json_dumps({'rows': self.rows, 'columns': self.columns.values()})
 
+    def merge(self, set):
+        self.rows = self.rows + set.rows
 
 def parse_issue(issue, field_mapping):
     result = OrderedDict()
@@ -179,6 +181,19 @@ def run_query(self, query, user):
                 results = parse_count(data)
             else:
                 results = parse_issues(data, field_mapping)
+                index = data['startAt'] + data['maxResults']
+
+                while data['total'] > index:
+                    query['startAt'] = index
+                    response, error = self.get_response(jql_url, params=query)
+                    if error is not None:
+                        return None, error
+
+                    data = response.json()
+                    index = data['startAt'] + data['maxResults']
+
+                    addl_results = parse_issues(data, field_mapping)
+                    results.merge(addl_results)
 
             return results.to_json(), None
         except KeyboardInterrupt:


Test patch: