Repository: OpenFn/adaptors
Pull request #1124: Improve auto-pagination in kobotoolbox
Author: login: mtuchi
Created: 2025-04-08T18:38:35Z | Merged: 2025-05-30T13:16:57Z

## Summary

This PR improve auto-pagination for `getSubmission()` by using it's own `util.requestWithPagination()` helper. This separates the paging behavior for `http.{get,post}`  helpers which uses `util.request()` function that does not support auto-paging.

This PR also adds a request operation

Fixes #988 #1191

## Details

- Update chai package to the latest version. 5.2.0
- Rename `Utils.js` to `util.js`
- Remove http export from common
- Add `maxRedirections` option in `HTTPRequesOptions`
- Add `util.requestWithPagination()` function
- Update `getSubmissions()` to use `util.requestWithPagination()` function
- Update `getSubmissions()` options for pagination 
- Update `getSumissions()` docs with examples
- Add few integration tests
- Update unit tests
- Log query params 


## AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to
know!):

- [ ] Code generation (copilot but not intellisense)
- [ ] Learning or fact checking
- [ ] Strategy / design
- [ ] Optimisation / refactoring
- [ ] Translation / spellchecking / doc gen
- [ ] Other
- [x] I have not used AI

You can read more details in our
[Responsible AI Policy](https://www.openfn.org/ai#pull-request-templates)

## Review Checklist

Before merging, the reviewer should check the following items:

- [x] Does the PR do what it claims to do?
- NA: If this is a new adaptor, added the adaptor on marketing website ?
- [ ] If this PR includes breaking changes, do we need to update any jobs in
      production? Is it safe to release?
- [x] Are there any unit tests?
- [x] Is there a changeset associated with this PR? Should there be? Note that
      dev only changes don't need a changeset.
- [x] Have you ticked a box under AI Usage?


Linked issue #988: kobotoolbox: improve pagination
kobotoolbox uses a fairly simple pagination right now. Basically `getSubmissions()` will always download everything, and the `http` helpers don't paginate at all.

What we want to support is:
- Add `limit`, `pageSize` and `start` options to any APIs that support pagination.
- Paginating APIs should download data until the `limit` is reached. If no limit, all data will be downloaded (I think this is fine for default behaviour BUT we should warn about it)
- I think only `getSubmissions()` and `http.get()` actually need to support this level of pagination.

Diff:
diff --git a/packages/gmail/CHANGELOG.md b/packages/gmail/CHANGELOG.md
index a09ae3010..73e6542e7 100644
--- a/packages/gmail/CHANGELOG.md
+++ b/packages/gmail/CHANGELOG.md
@@ -1,6 +1,6 @@
 # @openfn/language-gmail
 
-## 1.3.0
+## 1.3.0 - 30 May 2025
 
 ### Minor Changes
 
diff --git a/packages/kobotoolbox/CHANGELOG.md b/packages/kobotoolbox/CHANGELOG.md
index 16fee9d18..7cf625081 100644
--- a/packages/kobotoolbox/CHANGELOG.md
+++ b/packages/kobotoolbox/CHANGELOG.md
@@ -1,5 +1,54 @@
 # @openfn/language-kobotoolbox
 
+## 4.0.0 - 30 May 2025
+
+### Major Changes
+
+- fe5b899: Add automatic pagination to `getSubmissions()`.
+
+  `getSubmissions()` will now download all submissions, up to the requested
+  limit, over multiple requests. Note that by default, a maximum of 30,000 items
+  will be downloaded (pass a higher number of `Infinity` to download more).
+
+  ### Changes
+
+  - Updated `getSubmission` pagination behavior:
+
+    - Removed `paginate` option
+    - Added new pagination options:
+      - `limit`: Controls number of results returned
+      - `pageSize`: Number of items per page
+
+  - Remove export of `http` methods from `@openfn/language-common`
+
+  - Add support for `maxRedirections` option in `HTTPRequestOptions`
+
+  ### Migration Guide
+
+  1. Pagination Changes:
+
+     - Replace `paginate: true/false` with `limit` and `pageSize` options
+     - Default limit is `30000` unless otherwise specified
+
+     ```js
+     // Fetch submissions with default limit (30000)
+     getSubmission('form-id');
+
+     // Fetch all submissions (no limit)
+     getSubmission('form-id', { limit: Infinity });
+
+     // Fetch submissions with max results
+     getSubmission('form-id', { limit: 500 }); // Returns up to 500 submissions
+     ```
+
+  2. HTTP Changes:
+
+  - Support for `maxRedirections` option in `HTTPRequestOptions`
+    ```js
+    // Auto follow redirects
+    http.get('assets/aXecHjmbATuF6iGFmvBLBX/data', { maxRedirections: 5 });
+    ```
+
 ## 3.0.5 - 22 April 2025
 
 ### Patch Changes
diff --git a/packages/kobotoolbox/ast.json b/packages/kobotoolbox/ast.json
index 5bc1fc702..ba03d40ce 100644
--- a/packages/kobotoolbox/ast.json
+++ b/packages/kobotoolbox/ast.json
@@ -43,16 +43,21 @@
         "options"
       ],
       "docs": {
-        "description": "Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.",
+        "description": "Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`",
         "tags": [
           {
             "title": "example",
-            "description": "getSubmissions('aXecHjmbATuF6iGFmvBLBX');",
+            "description": "getSubmissions('aXecHjmbATuF6iGFmvBLBX');.",
+            "caption": "Get submissions for a specific form"
+          },
+          {
+            "title": "example",
+            "description": "getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });",
             "caption": "Get all submissions for a specific form"
           },
           {
             "title": "example",
-            "description": "getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: \"2022-06-12T21:54:20\" } } });",
+            "description": "getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: \"2025-03-12T21:54:20\" } } });",
             "caption": "Get form submissions with a query"
           },
           {
@@ -76,7 +81,7 @@
           },
           {
             "title": "param",
-            "description": "Optional query params for the request",
+            "description": "Options to control the request",
             "type": {
               "type": "OptionalType",
               "expression": {
@@ -87,6 +92,44 @@
             "name": "options",
             "default": "{}"
           },
+          {
+            "title": "param",
+            "description": "Query options to filter the submissions. See query operators {@link http://docs.mongodb.org/manual/reference/operator/query/.}",
+            "type": {
+              "type": "OptionalType",
+              "expression": {
+                "type": "NameExpression",
+                "name": "object"
+              }
+            },
+            "name": "options.query"
+          },
+          {
+            "title": "param",
+            "description": "Maximum number of submissions to fetch. Pass Infinity to disable the limit and download all submissions",
+            "type": {
+              "type": "OptionalType",
+              "expression": {
+                "type": "NameExpression",
+                "name": "number"
+              }
+            },
+            "name": "options.limit",
+            "default": "30000"
+          },
+          {
+            "title": "param",
+            "description": "Limits the size of each page of submissions. Maximum value is 30000.",
+            "type": {
+              "type": "OptionalType",
+              "expression": {
+                "type": "NameExpression",
+                "name": "number"
+              }
+            },
+            "name": "options.pageSize",
+            "default": "10000"
+          },
           {
             "title": "state",
             "description": "data - an array of submission objects"
@@ -101,7 +144,7 @@
           }
         ]
       },
-      "valid": true
+      "valid": false
     },
     {
       "name": "getDeploymentInfo",
diff --git a/packages/kobotoolbox/package.json b/packages/kobotoolbox/package.json
index d8a2e32e0..99f14662f 100644
--- a/packages/kobotoolbox/package.json
+++ b/packages/kobotoolbox/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@openfn/language-kobotoolbox",
   "label": "KoboToolbox",
-  "version": "3.0.5",
+  "version": "4.0.0",
   "description": "A KoboToolbox Language Pack for OpenFn",
   "homepage": "https://docs.openfn.org",
   "repository": {
@@ -11,8 +11,9 @@
   "main": "dist/index.cjs",
   "scripts": {
     "build": "pnpm clean && build-adaptor kobotoolbox",
-    "test": "mocha --experimental-specifier-resolution=node --no-warnings",
-    "test:watch": "mocha -w --experimental-specifier-resolution=node --no-warnings",
+    "test": "mocha --experimental-specifier-resolution=node --no-warnings --exclude test/integration.js --recursive",
+    "test:watch": "mocha -w --experimental-specifier-resolution=node --no-warnings --exclude test/integration.js --recursive",
+    "test:integration": "mocha --experimental-specifier-resolution=node --no-warnings test/integration.js",
     "clean": "rimraf dist types docs",
     "pack": "pnpm pack --pack-destination ../../dist",
     "lint": "eslint src"
@@ -30,7 +31,7 @@
   },
   "devDependencies": {
     "assertion-error": "^1.0.1",
-    "chai": "^3.4.0",
+    "chai": "^5.2.0",
     "deep-eql": "^0.1.3",
     "rimraf": "^3.0.2"
   },
@@ -44,4 +45,4 @@
     },
     "./package.json": "./package.json"
   }
-}
+}
\ No newline at end of file
diff --git a/packages/kobotoolbox/src/Adaptor.js b/packages/kobotoolbox/src/Adaptor.js
index e4d396fcf..d7967ffd6 100644
--- a/packages/kobotoolbox/src/Adaptor.js
+++ b/packages/kobotoolbox/src/Adaptor.js
@@ -4,15 +4,7 @@ import {
 } from '@openfn/language-common';
 import { expandReferences } from '@openfn/language-common/util';
 
-import * as util from './Utils';
-
-/**
- * Options object
- * @typedef {Object} RequestOptions
- * @property {object} query - An object of query parameters to be encoded into the URL
- * @property {object} headers - An object of all request headers
- * @property {string} [parseAs='json'] - The response format to parse (e.g., 'json', 'text', or 'stream')
- */
+import * as util from './util';
 
 /**
  * Execute a sequence of operations.
@@ -51,53 +43,65 @@ export function execute(...operations) {
  */
 export function getForms() {
   return async state => {
-    const url = `/assets/?asset_type=survey`;
+    const response = await util.request(state, 'GET', 'assets', {
+      query: { asset_type: 'survey' },
+    });
 
-    const response = await util.request(state, 'GET', url, {});
+    console.log('✓', response.body.results?.length, 'forms fetched.');
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
     return util.prepareNextState(state, response);
   };
 }
 
 /**
- * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
+ * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`
+ * @example <caption>Get submissions for a specific form</caption>
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX');.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
- * @param {object} [options={}] - Optional query params for the request
+ * @param {object} [options={}] - Options to control the request
+ * @param {object} [options.query] - Query options to filter the submissions. See query operators {@link http://docs.mongodb.org/manual/reference/operator/query/.}
+ * @param {number} [options.limit=30000] - Maximum number of submissions to fetch. Pass Infinity to disable the limit and download all submissions
+ * @param {number} [options.pageSize=10000] - Limits the size of each page of submissions. Maximum value is 30000.
  * @state data - an array of submission objects
  * @returns {Operation}
  */
-export function getSubmissions(formId, options = {}) {
+export function getSubmissions(formId, options) {
   return async state => {
-    const [resolvedFormId, resolvedOptions] = expandReferences(
+    const [resolvedFormId, resolvedOptions = {}] = expandReferences(
       state,
       formId,
       options
     );
 
-    const url = `/assets/${resolvedFormId}/data/`;
-    const query = {};
-    if (resolvedOptions.query) {
-      if (typeof resolvedOptions.query == 'string') {
-        query.query = resolvedOptions.query;
+    const { query, limit, pageSize } = resolvedOptions;
+    const path = `/assets/${resolvedFormId}/data/`;
+    const qs = {};
+    if (query) {
+      if (typeof query === 'string') {
+        qs.query = query;
       } else {
-        query.query = JSON.stringify(resolvedOptions.query);
+        qs.query = JSON.stringify(query);
       }
     }
+    const requestOptions = {
+      query: { ...qs },
+      limit,
+      pageSize,
+    };
+    const result = await util.requestWithPagination(
+      state,
+      path,
+      requestOptions
+    );
 
-    const { results } = await util.request(state, 'GET', url, {
-      paginate: true,
-      query,
-    });
-
-    console.log('✓', results.length, 'submissions fetched.');
-    return composeNextState(state, results);
+    console.log('✓', result?.length, 'submissions fetched.');
+    return composeNextState(state, result);
   };
 }
 
@@ -133,7 +137,6 @@ export {
   fields,
   fn,
   fnIf,
-  http,
   group,
   lastReferenceValue,
   merge,
diff --git a/packages/kobotoolbox/src/Utils.js b/packages/kobotoolbox/src/Utils.js
deleted file mode 100644
index 9e2f518cf..000000000
--- a/packages/kobotoolbox/src/Utils.js
+++ /dev/null
@@ -1,90 +0,0 @@
-import { composeNextState } from '@openfn/language-common';
-import {
-  request as commonRequest,
-  makeBasicAuthHeader,
-  logResponse,
-} from '@openfn/language-common/util';
-
-export const prepareNextState = (state, response) => {
-  const { body, ...responseWithoutBody } = response;
-
-  return {
-    ...composeNextState(state, response.body),
-    response: responseWithoutBody,
-  };
-};
-
-export async function request(state, method, path, opts) {
-  const { baseURL, apiVersion, username, password } = state.configuration;
-
-  let baseUrl;
-
-  if (!state.configuration.baseUrl && baseURL) {
-    baseUrl = baseURL;
-    console.warn(
-      'No baseUrl found in state.configuration. baseURL will be used instead, but this will be deprecated in the future.'
-    );
-  } else {
-    baseUrl = state.configuration.baseUrl;
-  }
-
-  const {
-    data = {},
-    query = {},
-    headers = {},
-    parseAs = 'json',
-    paginate = false,
-  } = opts;
-
-  const authHeaders = makeBasicAuthHeader(username, password);
-
-  let start, limit;
-
-  const options = {
-    body: data,
-    headers: {
-      'Content-Type': 'application/json',
-      ...authHeaders,
-      ...headers,
-    },
-    query: {
-      format: 'json',
-      ...query,
-    },
-    parseAs,
-    baseUrl: `${baseUrl}/api/${apiVersion}`,
-  };
-
-  if (paginate) {
-    const results = [];
-
-    do {
-      const response = await commonRequest(method, path, options).then(
-        logResponse
-      );
-      results.push(...response.body.results);
-      if (response.body.next) {
-        const nextUrl = new URL(response.body.next);
-
-        const startDigit =
-          nextUrl.searchParams.get('start') !== null
-            ? nextUrl.searchParams.get('start')
-            : 0;
-
-        start = Number(startDigit);
-        limit = nextUrl.searchParams.get('limit');
-
-        options.query = {
-          ...options.query,
-          start,
-          limit,
-        };
-      } else {
-        break;
-      }
-    } while (true);
-    return { results };
-  } else {
-    return commonRequest(method, path, options).then(logResponse);
-  }
-}
diff --git a/packages/kobotoolbox/src/http.js b/packages/kobotoolbox/src/http.js
index ccdeabd2f..69e605e12 100644
--- a/packages/kobotoolbox/src/http.js
+++ b/packages/kobotoolbox/src/http.js
@@ -1,38 +1,73 @@
 import { expandReferences } from '@openfn/language-common/util';
-import * as util from './Utils';
+import * as util from './util';
 
 /**
  * State object
- * @typedef {Object} KoboToolboxHttpState
+ * @typedef {Object} HttpState
+ * @private
  * @property data - The response body (as JSON)
- * @property response - The HTTP response from the KoboToolbox server (excluding the body). Responses will be returned in JSON format
- * @property references - An array of all previous data objects used in the Job
+ * @property response - The HTTP response from the KoboToolbox server (excluding the body)
+ * @property references - An array containing all previous data objects
  */
 
 /**
  * Options object
- * @typedef {Object} RequestOptions
+ * @typedef {Object} HTTPRequestOptions
  * @property {object} query - An object of query parameters to be encoded into the URL
  * @property {object} headers - An object of all request headers
+ * @property {object} body - The request body (as JSON)
+ * @property {number} maxRedirections - The maximum number of redirects to follow
  * @property {string} [parseAs='json'] - The response format to parse (e.g., 'json', 'text', or 'stream')
  */
 
+/**
+ * Make a HTTP request to any KoboToolbox endpoint
+ * @example <caption>Bulk updating of submissions</caption>
+ * http.request("PATCH", `assets/${$.form_uid}/data/bulk/`, {
+ *   body: {
+ *     submission_ids: [$.data.submission_id],
+ *     data: {
+ *       Transaction_status: "success",
+ *     },
+ *   },
+ * });
+ * @function
+ * @public
+ * @param {string} method - HTTP method to use
+ * @param {string} path - Path to resource
+ * @param {HTTPRequestOptions}  [options={}] - An object containing query, headers, and body for the request
+ * @state {HttpState}
+ * @returns {Operation}
+ */
+export function request(method, path, options = {}) {
+  return async state => {
+    const [resolvedMethod, resolvedPath, resolvedOptions = {}] =
+      expandReferences(state, method, path, options);
+
+    const response = await util.request(
+      state,
+      resolvedMethod,
+      resolvedPath,
+      resolvedOptions
+    );
+
+    return util.prepareNextState(state, response);
+  };
+}
 /**
  * Make a GET request to any KoboToolbox endpoint.
  * @public
  * @function
  * @example <caption>GET assets resource</caption>
- * http.get(
- *  "/assets/",
- *  )
+ * http.get('assets')
  * @param {string} path - path to resource
- * @param {RequestOptions} [options={}] - An object containing query params and headers for the request
- * @state {KoboToolboxHttpState}
+ * @param {HTTPRequestOptions} [options={}] - An object containing query params and headers for the request
+ * @state {HttpState}
  * @returns {operation}
  */
-export function get(path, options = {}) {
+export function get(path, options) {
   return async state => {
-    const [resolvedPath, resolvedOptions] = expandReferences(
+    const [resolvedPath, resolvedOptions = {}] = expandReferences(
       state,
       path,
       options
@@ -63,13 +98,13 @@ export function get(path, options = {}) {
  *  );
  * @param {string} path - path to resource
  * @param {any} data - the body data in JSON format
- * @param {RequestOptions} [options={}] - An object containing query params and headers for the request
- * @state {KoboToolboxHttpState}
+ * @param {HTTPRequestOptions} [options={}] - An object containing query params and headers for the request
+ * @state {HttpState}
  * @returns {operation}
  */
-export function post(path, data, options = {}) {
+export function post(path, data, options) {
   return async state => {
-    const [resolvedPath, resolvedData, resolvedOptions] = expandReferences(
+    const [resolvedPath, resolvedData, resolvedOptions = {}] = expandReferences(
       state,
       path,
       data,
@@ -105,13 +140,13 @@ export function post(path, data, options = {}) {
  *  );
  * @param {string} path - path to resource
  * @param {any} data - the body data in JSON format
- * @param {RequestOptions} [options={}] - An object containing query params and headers for the request
- * @state {KoboToolboxHttpState}
+ * @param {HTTPRequestOptions} [options={}] - An object containing query params and headers for the request
+ * @state {HttpState}
  * @returns {operation}
  */
-export function put(path, data, options = {}) {
+export function put(path, data, options) {
   return async state => {
-    const [resolvedPath, resolvedData, resolvedOptions] = expandReferences(
+    const [resolvedPath, resolvedData, resolvedOptions = {}] = expandReferences(
       state,
       path,
       data,
diff --git a/packages/kobotoolbox/src/util.js b/packages/kobotoolbox/src/util.js
new file mode 100644
index 000000000..1e1050a78
--- /dev/null
+++ b/packages/kobotoolbox/src/util.js
@@ -0,0 +1,115 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export const DEFAULT_LIMIT = 3e4;
+export const DEFAULT_PAGE_SIZE = 1e4;
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export async function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const {
+    data = {},
+    query = {},
+    headers = {},
+    parseAs = 'json',
+    maxRedirections,
+  } = opts;
+
+  const requestPath = `/api/${apiVersion}/${path}/`;
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    maxRedirections,
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
+}
+
+export async function requestWithPagination(state, path, options = {}) {
+  const results = [];
+
+  let { pageSize = DEFAULT_PAGE_SIZE, start, limit, ...otherOptions } = options;
+
+  const maxResults = limit ?? DEFAULT_LIMIT;
+
+  let isFirstRequest = true;
+  let requestOptions = { query: { start, limit: pageSize }, ...otherOptions };
+  let shouldFetchMoreContent = false;
+  const didUserPassLimit = Boolean(limit);
+
+  do {
+    requestOptions.query ??= {};
+    if (start) {
+      requestOptions.query.start = start;
+    }
+
+    if (didUserPassLimit || !isFirstRequest) {
+      // If there's an explicit limit or page size,
+      // or this is not the first request,
+      // set the limit in the URL
+      requestOptions.query.limit = Math.min(
+        pageSize || maxResults,
+        maxResults - results.length
+      );
+    } else if (!isNaN(pageSize)) {
+      requestOptions.query.limit = pageSize;
+    }
+
+    const response = await request(state, 'GET', path, requestOptions);
+
+    if (response.body?.results) {
+      results.push(...response.body.results);
+
+      if (response?.body?.next) {
+        const nextUrl = new URL(response?.body?.next);
+        start = nextUrl.searchParams.get('start');
+      }
+    } else {
+      results.push(response.body);
+    }
+
+    if (isFirstRequest && !pageSize) {
+      pageSize = results.length;
+    }
+
+    isFirstRequest = false;
+
+    const hasMoreContent = response.body.next?.includes('start=');
+
+    // If the user hasn't set a max but we've hit the limit, we should warn them
+    if (hasMoreContent && didUserPassLimit && results.length === maxResults) {
+      console.warn(
+        `Warning: The default maximum number of items has been reached (${maxResults}), but more items are available on the server. To download all available items, make another request with start=${
+          maxResults + 1
+        } or set max to Infinity`
+      );
+    }
+
+    // Decide whether to request another page
+    shouldFetchMoreContent = results.length < maxResults && hasMoreContent;
+  } while (shouldFetchMoreContent);
+
+  return results;
+}
diff --git a/packages/kobotoolbox/test/ClientFixtures.js b/packages/kobotoolbox/test/ClientFixtures.js
deleted file mode 100644
index c79d89602..000000000
--- a/packages/kobotoolbox/test/ClientFixtures.js
+++ /dev/null
@@ -1,44 +0,0 @@
-const fixtures = {
-  post: {
-    requestBody: {
-      "A": 1
-    },
-    responseBody: {
-      "httpStatus": "OK",
-      "httpStatusCode": 200,
-      "status": "OK",
-      "message": "Import was successful.",
-      "response": {
-        "responseType": "ImportSummaries",
-        "imported": 1,
-        "updated": 0,
-        "deleted": 0,
-        "ignored": 0
-      }
-    }
-  }
-}
-
-export {fixtures};
-
-export default [
-  {
-    pattern : 'https://fake.server.com/api(.*)',
-
-    fixtures(match, params, headers) {
-      if (match[1] === '/api') {
-        return {body: fixtures.post.responseBody, params, headers}
-      }
-
-      throw new Error(`No Fixture Match\ngot: ${JSON.stringify(match, 2, null)}`)
-    },
-
-    post(match, data) {
-      return {
-        ok: true,
-        match,
-        ...data
-      }
-    }
-  }
-]
diff --git a/packages/kobotoolbox/test/adaptor.test.js b/packages/kobotoolbox/test/adaptor.test.js
new file mode 100644
index 000000000..abd150bcc
--- /dev/null
+++ b/packages/kobotoolbox/test/adaptor.test.js
@@ -0,0 +1,335 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_PAGE_SIZE, DEFAULT_LIMIT } from '../src/util';
+const defaultLimit = DEFAULT_LIMIT;
+const pageLimit = DEFAULT_PAGE_SIZE;
+
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleData = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+
+  it('should not return more items than the default limit', async () => {
+    let requestCount = 0;
+    const items = Array.from({ length: 4e4 }, (_, i) => ({
+      uid: String(i),
+    }));
+
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            items,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(3);
+
+    const { data } = await execute(getSubmissions('aXecHjmbATuF6iGFmvBLBX'))(
+      state
+    );
+
+    expect(requestCount).to.eql(3);
+    expect(data.length).to.eql(3e4);
+    expect(items.length).to.greaterThan(data.length);
+  });
+
+  it('should only return 1 item if the limit is 1', async () => {
+    const limit = 1;
+    const mockData = sampleData.slice(0, limit);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleData,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(1);
+
+    const { data } = await execute(
+      getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit })
+    )(state);
+
+    expect(data).to.eql(mockData);
+    expect(requestCount).to.eql(1);
+    expect(data.length).to.eql(1);
+  });
+
+  it('should be able to explicity get more items than the default max limit', async () => {
+    const manyObjects = Array.from({ length: defaultLimit + 2 }, (_, i) => ({
+      uid: String(i),
+    }));
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            manyObjects,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(4);
+
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX', {
+      limit: 4e4,
+    })(state);
+
+    expect(requestCount).to.eql(4);
+    expect(data.length).to.eql(3e4 + 2);
+    expect(data.length).to.greaterThan(defaultLimit);
+  });
+  it('should allow custom pageSize', async () => {
+    const state = { configuration };
+    const pageSize = 1;
+    const mockData = Array.from({ length: 10 }, (_, i) => ({
+      uid: String(i),
+    }));
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            mockData,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+              defaultLimit: 1,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(10);
+
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX', {
+      pageSize,
+    })(state);
+
+    expect(data.length).to.eql(10);
+    expect(requestCount).to.eql(10);
+  });
+
+  it('should return all items if limit is Infinity', async () => {
+    const mockData = Array.from({ length: defaultLimit + 1 }, (_, i) => ({
+      uid: String(i),
+    }));
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            mockData,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(4);
+
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX', {
+      limit: Infinity,
+    })(state);
+
+    expect(data.length).to.eql(3e4 + 1);
+    expect(requestCount).to.eql(4);
+  });
+
+  afterEach(() => {
+    console.warn = originalLog;
+  });
+});
+
+describe('getForms', () => {
+  beforeEach(() => {
+    testServer
+      .intercept({
+        path: `/api/v2/assets/`,
+        method: 'GET',
+        query: { format: 'json', asset_type: 'survey' },
+      })
+      .reply(
+        200,
+        { results: [{ name: 'Feedback Survey Test', asset_type: 'survey' }] },
+        { ...jsonHeaders }
+      );
+  });
+
+  it('should get a list of forms', async () => {
+    const state = { configuration };
+    const { data } = await execute(getForms())(state);
+    expect(data.results[0].name).to.eql('Feedback Survey Test');
+  });
+});
+
+describe('getDeploymentInfo', () => {
+  const formId = 'aXecHjmbATuF6iGFmvBLBX';
+
+  beforeEach(() => {
+    testServer
+      .intercept({
+        path: `/api/v2/assets/${formId}/deployment/`,
+        method: 'GET',
+        query: { format: 'json' },
+      })
+      .reply(
+        200,
+        { asset: { name: 'Feedback Survey Test', asset_type: 'survey' } },
+        { ...jsonHeaders }
+      );
+  });
+
+  it('should get a list of deployment', async () => {
+    const state = { configuration };
+    const { data } = await execute(getDeploymentInfo(formId))(state);
+    expect(data.asset.name).to.eql('Feedback Survey Test');
+  });
+});
+
+describe('execute', () => {
+  it('executes each operation in sequence', done => {
+    let state = {};
+    let operations = [
+      state => {
+        return { counter: 1 };
+      },
+      state => {
+        return { counter: 2 };
+      },
+      state => {
+        return { counter: 3 };
+      },
+    ];
+
+    execute(...operations)(state)
+      .then(finalState => {
+        expect(finalState).to.eql({ counter: 3 });
+      })
+      .then(done)
+      .catch(done);
+  });
+
+  it('assigns references, data to the initialState', () => {
+    let state = {};
+
+    let finalState = execute()(state);
+
+    execute()(state).then(finalState => {
+      expect(finalState).to.eql({ references: [], data: null });
+    });
+  });
+});
diff --git a/packages/kobotoolbox/test/helper.js b/packages/kobotoolbox/test/helper.js
new file mode 100644
index 000000000..ed4a98dc9
--- /dev/null
+++ b/packages/kobotoolbox/test/helper.js
@@ -0,0 +1,65 @@
+import { DEFAULT_PAGE_SIZE, DEFAULT_LIMIT } from '../src/util';
+
+export function responseWithPagination(items, query, settings) {
+  const {
+    url,
+    pageLimit = DEFAULT_PAGE_SIZE,
+    defaultLimit = DEFAULT_LIMIT,
+  } = settings;
+  const { start = 0, limit } = query;
+
+  const response = {
+    results: [],
+    next: null,
+    previous: null,
+    count: items.length,
+    requestCount: 0,
+  };
+
+  let startIndex = parseInt(start);
+  const hasStartIndex = startIndex > 0;
+  let shouldFetchMoreContent = false;
+  let effectiveLimit = limit ?? pageLimit;
+  let isUsingDefaultLimit = !limit && pageLimit === DEFAULT_PAGE_SIZE;
+
+  let maxResults = defaultLimit;
+  const hasNoLimitWithStartIndex = !limit && hasStartIndex;
+  if (hasNoLimitWithStartIndex) {
+    maxResults = items.length - startIndex;
+  }
+
+  do {
+    response.results.push(
+      ...items.slice(startIndex, startIndex + effectiveLimit)
+    );
+    const fetchedAll =
+      response.results.length ===
+      (limit < items.length ? effectiveLimit : items.length);
+
+    const reachedMaxLimit = response.results.length >= maxResults;
+
+    shouldFetchMoreContent = !fetchedAll && !reachedMaxLimit && false;
+
+    response.requestCount++;
+    startIndex += effectiveLimit;
+
+    response.next =
+      startIndex >= items.length ||
+      limit > items.length || // If there are less items than the limit, return all and no next link
+      (!limit && !hasStartIndex && fetchedAll) || // If there are no items, start = 0 and no limit, return all and no next link
+      (fetchedAll && hasNoLimitWithStartIndex) // If there are more items, no limit and start > 0, return all and no next link
+        ? null
+        : `${url}?format=json&start=${startIndex}&limit=${effectiveLimit}`;
+
+    response.previous =
+      !hasStartIndex && startIndex <= effectiveLimit
+        ? null
+        : hasNoLimitWithStartIndex && isUsingDefaultLimit
+        ? `${url}?format=json&limit=${defaultLimit}` // If no limit and start >0
+        : `${url}?format=json&start=${Math.max(
+            0,
+            startIndex - effectiveLimit
+          )}&limit=${effectiveLimit}`;
+  } while (shouldFetchMoreContent);
+  return response;
+}
diff --git a/packages/kobotoolbox/test/http.test.js b/packages/kobotoolbox/test/http.test.js
new file mode 100644
index 000000000..8821f4912
--- /dev/null
+++ b/packages/kobotoolbox/test/http.test.js
@@ -0,0 +1,87 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import { http } from '../src';
+
+const testServer = enableMockClient('https://http-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://http-test.kobotoolbox.org',
+};
+describe('http.get', () => {
+  it('should GET with a query', async () => {
+    testServer
+      .intercept({
+        path: '/api/v2/assets/',
+        query: { format: 'json', metadata: 'on' },
+        method: 'GET',
+      })
+      .reply(
+        200,
+        {
+          results: [{ name: 'Feedback Survey Test', asset_type: 'survey' }],
+        },
+        { ...jsonHeaders }
+      );
+    const state = { configuration };
+
+    const { response, data } = await http.get('/assets/', {
+      query: { metadata: 'on' },
+    })(state);
+    expect(data.results[0].name).to.eql('Feedback Survey Test');
+    expect(response.headers['content-type']).to.eql('application/json');
+  });
+
+  it('should make a GET request', async () => {
+    testServer
+      .intercept({
+        path: '/api/v2/assets/',
+        query: { format: 'json' },
+        method: 'GET',
+      })
+      .reply(
+        200,
+        { results: [{ name: 'Feedback Survey Test', asset_type: 'survey' }] },
+        { ...jsonHeaders }
+      );
+    const state = { configuration };
+
+    const { data } = await http.get('/assets/')(state);
+
+    expect(data.results[0].name).to.eql('Feedback Survey Test');
+  });
+});
+
+describe('http.post', () => {
+  it('should make a POST request', async () => {
+    testServer
+      .intercept({
+        path: '/api/v2/assets/',
+        query: { format: 'json' },
+        method: 'POST',
+        data: {
+          name: 'Feedback Survey Test',
+          asset_type: 'survey',
+        },
+      })
+      .reply(
+        200,
+        { name: 'Feedback Survey Test', asset_type: 'survey' },
+        { ...jsonHeaders }
+      );
+    const state = { configuration };
+
+    const { data } = await http.post('/assets/', {
+      name: 'Feedback Survey Test',
+      asset_type: 'survey',
+    })(state);
+
+    expect(data.name).to.eql('Feedback Survey Test');
+  });
+});
diff --git a/packages/kobotoolbox/test/index.js b/packages/kobotoolbox/test/index.js
deleted file mode 100644
index 344837371..000000000
--- a/packages/kobotoolbox/test/index.js
+++ /dev/null
@@ -1,373 +0,0 @@
-import chai from 'chai';
-const { expect } = chai;
-import { enableMockClient } from '@openfn/language-common/util';
-import { request } from '../src/Utils';
-
-import { http } from '../src';
-
-import {
-  getSubmissions,
-  getForms,
-  getDeploymentInfo,
-  execute,
-} from '../src/Adaptor';
-
-const testServer = enableMockClient('https://test.kobotoolbox.org');
-const jsonHeaders = {
-  headers: {
-    'Content-Type': 'application/json',
-  },
-};
-const configuration = {
-  username: 'test',
-  password: 'strongpassword',
-  apiVersion: 'v2',
-  baseUrl: 'https://test.kobotoolbox.org',
-};
-
-const defaultObjects = [
-  { uid: '1' },
-  { uid: '2' },
-  { uid: '3' },
-  { uid: '4' },
-  { uid: '5' },
-  { uid: '6' },
-];
-
-const paginatedResponse = (
-  url,
-  limit = 30000,
-  start = 0,
-  objects = defaultObjects
-) => {
-  const s = parseInt(start);
-  const l = parseInt(limit);
-  const results = objects.slice(s, s + l);
-  if (objects.length > s + l) {
-    const next = `${url}/?format=json&start=${s + l}&limit=${l}`;
-
-    return {
-      next,
-      results,
-    };
-  } else {
-    return { results };
-  }
-};
-
-describe('paginatedResponse', () => {
-  const items = [1, 2, 3];
-  it('should return the first item', () => {
-    const start = 0;
-    const limit = 1;
-    const { next, results } = paginatedResponse('www', limit, start, items);
-    expect(results).to.eql([1]);
-  });
-  it('should return 2 items with limit 2', () => {
-    const start = 0;
-    const limit = 2;
-    const { next, results } = paginatedResponse('www', limit, start, items);
-    expect(results).to.eql([1, 2]);
-  });
-  it('should return 2 items with start 1 and limit 2', () => {
-    const start = 1;
-    const limit = 2;
-    const { next, results } = paginatedResponse('www', limit, start, items);
-    expect(results).to.eql([2, 3]);
-  });
-  it('should return all items', () => {
-    const start = 0;
-    const limit = 3;
-    const { next, results } = paginatedResponse('www', limit, start, items);
-    expect(results).to.eql([1, 2, 3]);
-  });
-  it('should return all items if limit is greater', () => {
-    const start = 0;
-    const limit = 100;
-    const { next, results } = paginatedResponse('www', limit, start, items);
-    expect(results).to.eql([1, 2, 3]);
-  });
-  it('should return the correct next link', () => {
-    const start = 1;
-    const limit = 1;
-    const { next, results } = paginatedResponse('www', limit, start, items);
-    expect(next).to.eql('www/?format=json&start=2&limit=1');
-    expect(results).to.eql([2]);
-  });
-});
-
-describe('execute', () => {
-  it('executes each operation in sequence', done => {
-    let state = {};
-    let operations = [
-      state => {
-        return { counter: 1 };
-      },
-      state => {
-        return { counter: 2 };
-      },
-      state => {
-        return { counter: 3 };
-      },
-    ];
-
-    execute(...operations)(state)
-      .then(finalState => {
-        expect(finalState).to.eql({ counter: 3 });
-      })
-      .then(done)
-      .catch(done);
-  });
-
-  it('assigns references, data to the initialState', () => {
-    let state = {};
-
-    let finalState = execute()(state);
-
-    execute()(state).then(finalState => {
-      expect(finalState).to.eql({ references: [], data: null });
-    });
-  });
-});
-
-describe('request', () => {
-  it('handles pagination if paginate is true', async () => {
-    let callCount = 0;
-    testServer
-      .intercept({
-        path: /\/api\/v2\/assets\/aDReHdA7UuNBYsiCXQBr43\/data\//,
-        method: 'GET',
-      })
-      .reply(
-        200,
-        req => {
-          callCount++;
-          return paginatedResponse(
-            `${req.origin}${req.path}`,
-            req.query.limit,
-            req.query.start
-          );
-        },
-        { ...jsonHeaders }
-      )
-      .times(6);
-    const state = { configuration };
-    const { results } = await request(
-      state,
-      'GET',
-      '/assets/aDReHdA7UuNBYsiCXQBr43/data/',
-      {
-        paginate: true,
-        query: { start: 0, limit: 1 },
-      }
-    );
-
-    expect(callCount).to.eql(6);
-    expect(results).to.eql(defaultObjects);
-  });
-  it('does not handle pagination if paginate is false', async () => {
-    let callCount = 0;
-    testServer
-      .intercept({
-        path: /\/api\/v2\/assets\/aDReHdA7UuNBYsiCXQBr43\/data\//,
-        method: 'GET',
-      })
-      .reply(
-        200,
-        req => {
-          callCount++;
-          return paginatedResponse(
-            `${req.origin}${req.path}`,
-            req.query.limit,
-            req.query.start
-          );
-        },
-        { ...jsonHeaders }
-      )
-      .times(1);
-    const state = { configuration };
-    const { body } = await request(
-      state,
-      'GET',
-      '/assets/aDReHdA7UuNBYsiCXQBr43/data/',
-      {
-        paginate: false,
-      }
-    );
-
-    expect(callCount).to.eql(1);
-    expect(body.results).to.eql(defaultObjects);
-  });
-});
-
-describe('http.get', () => {
-  it('should GET with a query', async () => {
-    testServer
-      .intercept({
-        path: '/api/v2/assets/',
-        query: { format: 'json' },
-        method: 'GET',
-      })
-      .reply(
-        200,
-        {
-          results: [{ name: 'Feedback Survey Test', asset_type: 'survey' }],
-        },
-        { ...jsonHeaders }
-      );
-    const state = { configuration };
-
-    const response = await http.get('/assets/')(state);
-    expect(response.response.headers['content-type']).to.eql(
-      'application/json'
-    );
-  });
-});
-
-describe('http.get', () => {
-  it('should make a GET request', async () => {
-    testServer
-      .intercept({
-        path: '/api/v2/assets/',
-        query: { format: 'json' },
-        method: 'GET',
-      })
-      .reply(
-        200,
-        { results: [{ name: 'Feedback Survey Test', asset_type: 'survey' }] },
-        { ...jsonHeaders }
-      );
-    const state = { configuration };
-
-    const { data } = await http.get('/assets/')(state);
-
-    expect(data.results[0].name).to.eql('Feedback Survey Test');
-  });
-});
-
-describe('http.post', () => {
-  it('should make a POST request', async () => {
-    testServer
-      .intercept({
-        path: '/api/v2/assets/',
-        query: { format: 'json' },
-        method: 'POST',
-        data: {
-          name: 'Feedback Survey Test',
-          asset_type: 'survey',
-        },
-      })
-      .reply(
-        200,
-        { name: 'Feedback Survey Test', asset_type: 'survey' },
-        { ...jsonHeaders }
-      );
-    const state = { configuration };
-
-    const { data } = await http.post('/assets/', {
-      name: 'Feedback Survey Test',
-      asset_type: 'survey',
-    })(state);
-
-    expect(data.name).to.eql('Feedback Survey Test');
-  });
-});
-
-describe('getSubmissions', () => {
-  const formId = 'aXecHjmbATuF6iGFmvBLBX';
-
-  it('should get a list of submissions', async () => {
-    testServer
-      .intercept({
-        path: `/api/v2/assets/${formId}/data/`,
-        method: 'GET',
-        query: { format: 'json' },
-      })
-      .reply(
-        200,
-        {
-          results: [
-            { First_Name_of_Patient: 'Kwothe', Last_Name_of_Patient: 'Ruh' },
-          ],
-        },
-        { ...jsonHeaders }
-      );
-    const state = { configuration };
-    const { data } = await execute(getSubmissions(formId))(state);
-    expect(data[0]['First_Name_of_Patient']).to.eql('Kwothe');
-  });
-
-  it('should get a list of submissions with a query', async () => {
-    testServer
-      .intercept({
-        path: `/api/v2/assets/${formId}/data/`,
-        method: 'GET',
-        query: {
-          format: 'json',
-          query: '{"_submission_time":{"$gte":"2022-06-12T21:54:20"}}',
-        },
-      })
-      .reply(
-        200,
-        {
-          results: [
-            { First_Name_of_Patient: 'Kwothe', Last_Name_of_Patient: 'Ruh' },
-          ],
-        },
-        { ...jsonHeaders }
-      );
-    const state = { configuration };
-    const { data } = await execute(
-      getSubmissions(formId, {
-        query: { _submission_time: { $gte: '2022-06-12T21:54:20' } },
-      })
-    )(state);
-    expect(data[0]['First_Name_of_Patient']).to.eql('Kwothe');
-  });
-});
-
-describe('getForms', () => {
-  beforeEach(() => {
-    testServer
-      .intercept({
-        path: `/api/v2/assets/`,
-        method: 'GET',
-        query: { format: 'json', asset_type: 'survey' },
-      })
-      .reply(
-        200,
-        { results: [{ name: 'Feedback Survey Test', asset_type: 'survey' }] },
-        { ...jsonHeaders }
-      );
-  });
-
-  it('should get a list of forms', async () => {
-    const state = { configuration };
-    const { data } = await execute(getForms())(state);
-    expect(data.results[0].name).to.eql('Feedback Survey Test');
-  });
-});
-
-describe('getDeploymentInfo', () => {
-  const formId = 'aXecHjmbATuF6iGFmvBLBX';
-
-  beforeEach(() => {
-    testServer
-      .intercept({
-        path: `/api/v2/assets/${formId}/deployment/`,
-        method: 'GET',
-        query: { format: 'json' },
-      })
-      .reply(
-        200,
-        { asset: { name: 'Feedback Survey Test', asset_type: 'survey' } },
-        { ...jsonHeaders }
-      );
-  });
-
-  it('should get a list of deployment', async () => {
-    const state = { configuration };
-    const { data } = await execute(getDeploymentInfo(formId))(state);
-    expect(data.asset.name).to.eql('Feedback Survey Test');
-  });
-});
diff --git a/packages/kobotoolbox/test/integration.js b/packages/kobotoolbox/test/integration.js
new file mode 100644
index 000000000..6a710cadc
--- /dev/null
+++ b/packages/kobotoolbox/test/integration.js
@@ -0,0 +1,103 @@
+// For creds.json, See LP: OpenFn KoboToolBox Demo
+import { expect } from 'chai';
+import configuration from '../tmp/creds.json' assert { type: 'json' };
+import {
+  getDeploymentInfo,
+  getSubmissions,
+  getForms,
+  execute,
+  http,
+  fn,
+} from '../dist/index.js';
+
+const state = { configuration };
+
+describe('Integration tests', () => {
+  describe('http', () => {
+    describe('post', () => {
+      it('should post a request to kobo', async () => {
+        const { data, response } = await execute(
+          http.post('assets', {
+            name: 'Health Survey 2024',
+            settings: {
+              description:
+                'A comprehensive health survey for rural communities',
+              sector: 'Health',
+              country: 'Tanzania',
+              'share-metadata': true,
+            },
+            asset_type: 'survey',
+          })
+        )(state).catch(console.error);
+
+        expect(response.statusCode).to.eq(201);
+        expect(response.statusMessage).to.eq('Created');
+        expect(data).to.have.property('url');
+      }).timeout(5000);
+    });
+    describe('get', () => {
+      it('should throw an error if path is absolute', async () => {
+        try {
+          await execute(http.get('https://www.google.com'))(state);
+        } catch (error) {
+          expect(error.code).to.eq('ERR_INVALID_URL');
+        }
+      }).timeout(5000);
+
+      it('fetches submission data with a query', async () => {
+        const { data } = await execute(
+          http.get('/assets/aUe2eV8pHK9DUEUxT9rCcs/data/', {
+            query: {
+              limit: 3,
+            },
+          })
+        )(state);
+
+        expect(data.count).to.greaterThan(data.results.length);
+        expect(data.next).to.contain('?format=json&limit=3');
+        expect(data.results.length).to.eq(3);
+      }).timeout(5000);
+    });
+  });
+  describe('getForms', () => {
+    it('should get a list of forms', async () => {
+      state.data = {};
+      const { data } = await execute(getForms())(state);
+
+      expect(data).to.have.all.keys('count', 'next', 'previous', 'results');
+      expect(data).to.have.property('next', null);
+      expect(data).to.have.property('previous', null);
+    }).timeout(5000);
+  });
+
+  describe('getSubmissions', () => {
+    it('should get a list of submissions', async () => {
+      const { data } = await execute(
+        getSubmissions('aUe2eV8pHK9DUEUxT9rCcs', { pageSize: 3 })
+      )(state);
+
+      expect(data.length).to.greaterThan(3);
+    }).timeout(50000);
+
+    it('should get a list of submissions with a query', async () => {
+      const { data } = await execute(
+        getSubmissions('aUe2eV8pHK9DUEUxT9rCcs', {
+          query: { _submission_time: { $gte: '2025-04-04T21:54:20' } },
+        })
+      )(state);
+      expect(data.length).to.greaterThan(0);
+    });
+  });
+
+  describe('getDeploymentInfo', () => {
+    it('should get a list of deployment', async () => {
+      const { data } = await execute(
+        getDeploymentInfo('aUe2eV8pHK9DUEUxT9rCcs')
+      )(state);
+
+      expect(data.asset.name).to.eq(
+        'Ghana Bed-nets Inventory Tracking Questionare'
+      );
+    }).timeout(5000);
+  });
+});
diff --git a/packages/kobotoolbox/test/util.test.js b/packages/kobotoolbox/test/util.test.js
new file mode 100644
index 000000000..0d4aa2cd9
--- /dev/null
+++ b/packages/kobotoolbox/test/util.test.js
@@ -0,0 +1,272 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use the default limit if no limit is passed', () => {
+    const { results } = responseWithPagination(
+      items,
+      {},
+      { url: 'www', pageLimit: 1 }
+    );
+
+    expect(results).to.eql([1]);
+    expect(results.length).to.eql(1);
+  });
+  it('use the user limit, not the default limit', () => {
+    const { results } = responseWithPagination(
+      items,
+      { limit: 2 },
+      { url: 'www' }
+    );
+    expect(results).to.eql([1, 2]);
+    expect(results.length).to.eql(2);
+  });
+
+  it('pagination values are included when additional items exist', () => {
+    const limit = 2;
+    const { next, previous } = responseWithPagination(
+      items,
+      {
+        limit,
+      },
+      { url: 'www' }
+    );
+
+    expect(next).to.eql('www?format=json&start=2&limit=2');
+    expect(previous).to.eql(null);
+  });
+
+  it('should return first 100 items when pageLimit=100 for dataset of 1001 items', async () => {
+    const pageLimit = 100;
+    const { results, next, previous } = responseWithPagination(
+      Array.from({ length: 1001 }, (_, i) => ({
+        uid: String(i),
+      })),
+      {},
+      { url: 'www', pageLimit }
+    );
+    expect(results.length).to.eql(pageLimit);
+    expect(next).to.eql(
+      `www?format=json&start=${pageLimit}&limit=${pageLimit}`
+    );
+    expect(previous).to.eql(null);
+  });
+
+  it('should return 2 items with start 1 and pageLimit 2', () => {
+    const start = 1;
+    const pageLimit = 2;
+    const { results, next, previous } = responseWithPagination(
+      items,
+      {
+        start,
+      },
+      { url: 'www', pageLimit }
+    );
+
+    expect(results).to.eql([2, 3]);
+    expect(next).to.eql(null);
+    expect(previous).to.eql(`www?format=json&start=1&limit=${pageLimit}`);
+  });
+  it('should return all items if items do not exceed default limit', () => {
+    const limit = 3;
+    const { results } = responseWithPagination(
+      items,
+      { limit },
+      { url: 'www' }
+    );
+
+    expect(results).to.eql([1, 2, 3]);
+    expect(results.length).to.eql(3);
+    expect(results.length).to.lessThanOrEqual(limit);
+  });
+  it('should return first pageLimit items with next link when exceeding defaultLimit', async () => {
+    const defaultLimit = 3;
+    const pageLimit = 1;
+    const { results, next, count } = responseWithPagination(
+      Array.from({ length: defaultLimit + 1 }, (_, i) => ({
+        uid: String(i),
+      })),
+      {},
+      { url: 'www', pageLimit, defaultLimit }
+    );
+
+    expect(count).to.eql(4);
+    expect(results.length).to.eql(1);
+    expect(count).to.greaterThan(defaultLimit);
+    expect(results.length).to.eql(pageLimit);
+    expect(next).to.eql(`www?format=json&start=1&limit=1`);
+  });
+});
+
+describe('request', () => {
+  it('should make GET a request', async () => {
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aUe2eV8pHK9DUEUxT9rCcs\/data\/?/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        {
+          count: 1,
+          next: null,
+          previous: null,
+          results: [{ name: 'Feedback Survey Test', asset_type: 'survey' }],
+        },
+        jsonHeaders
+      );
+    const state = { configuration };
+    const response = await util.request(
+      state,
+      'GET',
+      '/assets/aUe2eV8pHK9DUEUxT9rCcs/data/'
+    );
+
+    expect(response.method).to.eq('GET');
+    expect(response.statusCode).to.eq(200);
+    expect(response.statusMessage).to.eq('OK');
+    expect(response.body.count).to.eq(response.body.results.length);
+  });
+});
+describe('requestWithPagination', () => {
+  it('should return only 1 item when limit=1 is specified', async () => {
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aDReHdA7UuNBYsiCXQBr23\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          return responseWithPagination(
+            defaultObjects,
+            { limit: req.query.limit, start: req.query.start },
+            { url: `${req.origin}${req.path}` }
+          );
+        },
+        { ...jsonHeaders }
+      )
+      .times(1);
+    const state = { configuration };
+    const data = await util.requestWithPagination(
+      state,
+      '/assets/aDReHdA7UuNBYsiCXQBr23/data/',
+      {
+        limit: 1,
+      }
+    );
+
+    expect(requestCount).to.eql(1);
+    expect(data).to.eql([{ uid: '1' }]);
+  });
+  it('should return all 6 items by making 6 separate API requests with pageSize=1', async () => {
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aDReHdA7UuNBYsiCXQBr43\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const response = responseWithPagination(
+            defaultObjects,
+            { start: req.query.start, limit: req.query.limit },
+            {
+              url: `${req.origin}${req.path}`,
+              pageLimit: 1,
+            }
+          );
+
+          return response;
+        },
+        { ...jsonHeaders }
+      )
+      .times(6);
+
+    const state = { configuration };
+    const data = await util.requestWithPagination(
+      state,
+      '/assets/aDReHdA7UuNBYsiCXQBr43/data/',
+      {
+        pageSize: 1,
+      }
+    );
+
+    expect(data.length).to.eql(6);
+    expect(requestCount).to.eql(6);
+    expect(data).to.eql(defaultObjects);
+  });
+  it('should return defaultLimit items and stop pagination, even when more items are available', async () => {
+    let requestCount = 0;
+    const defaultLimit = 3;
+    const defaulLimit = 1;
+    const mockData = Array.from({ length: defaultLimit + 1 }, (_, i) => ({
+      uid: String(i),
+    }));
+
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aDReHdA7UuNBYsiCXQBr33\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          return responseWithPagination(
+            mockData,
+            { limit: req.query.limit, start: req.query.start },
+            { url: `${req.origin}${req.path}` }
+          );
+        },
+        { ...jsonHeaders }
+      )
+      .times(3);
+
+    const state = { configuration };
+    const data = await util.requestWithPagination(
+      state,
+      '/assets/aDReHdA7UuNBYsiCXQBr33/data/',
+      {
+        limit: defaultLimit,
+        pageSize: defaulLimit,
+      }
+    );
+
+    expect(data.length).to.eql(3);
+    expect(requestCount).to.eql(3);
+    expect(mockData.length).to.greaterThan(data.length);
+  });
+});
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index b86b23947..16354c4ac 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -984,8 +984,8 @@ importers:
         specifier: ^1.0.1
         version: 1.1.0
       chai:
-        specifier: ^3.4.0
-        version: 3.5.0
+        specifier: ^5.2.0
+        version: 5.2.0
       deep-eql:
         specifier: ^0.1.3
         version: 0.1.3
@@ -5236,13 +5236,13 @@ packages:
     resolution: {integrity: sha512-MzMFlSLBqNF2gcHWO0G1vP/YQyfvrxZ0bF+u7mzUdZ1/xK4A4sru+nraZz5i3iEIk1l1uyicaDVTB4QbbEkAYg==}
     dependencies:
       '@types/eslint': 9.6.1
-      '@types/estree': 1.0.6
+      '@types/estree': 1.0.7
     dev: false
 
   /@types/eslint@9.6.1:
     resolution: {integrity: sha512-FXx2pKgId/WyYo2jXw63kk7/+TY7u7AziEJxJAnSFzHlqTAS3Ync6SvgYAN/k4/PQpnnVuzoMuVnByKK2qp0ag==}
     dependencies:
-      '@types/estree': 1.0.6
+      '@types/estree': 1.0.7
       '@types/json-schema': 7.0.15
     dev: false
 
@@ -5250,6 +5250,10 @@ packages:
     resolution: {integrity: sha512-AYnb1nQyY49te+VRAVgmzfcgjYS91mY5P0TKUDCLEM+gNnA+3T6rWITXRLYCpahpqSQbN5cE+gHpnPyXjHWxcw==}
     dev: false
 
+  /@types/estree@1.0.7:
+    resolution: {integrity: sha512-w28IoSUCJpidD/TGviZwwMJckNESJZXFu7NBZ5YJ4mEUnNraUn9Pm8HSZm/jDF1pDWYKspWE7oVphigUPRakIQ==}
+    dev: false
+
   /@types/fhir@0.0.41:
     resolution: {integrity: sha512-MAQAFufNZBZ6V0F94Nhknmmh/E3iMXFK4n/L8RkSNjKtOJnvaAJERivNOj35VVx9VCQBJbE0BHSzikfBahoRhA==}
     dev: false
@@ -7277,6 +7281,17 @@ packages:
       pathval: 2.0.0
     dev: true
 
+  /chai@5.2.0:
+    resolution: {integrity: sha512-mCuXncKXk5iCLhfhwTc0izo0gtEmpz5CtG2y8GiOINBlMVS6v8TMRc5TaLWKS6692m9+dVVfzgeVxR5UxWHTYw==}
+    engines: {node: '>=12'}
+    dependencies:
+      assertion-error: 2.0.1
+      check-error: 2.1.1
+      deep-eql: 5.0.2
+      loupe: 3.1.1
+      pathval: 2.0.0
+    dev: true
+
   /chainsaw@0.1.0:
     resolution: {integrity: sha512-75kWfWt6MEKNC8xYXIdRpDehRYY/tNSgwKaJq+dbbDcxORuVrrQ+SEHoWsniVn9XPYfP4gmdWIeDk/4YNp1rNQ==}
     dependencies:
@@ -15932,7 +15947,7 @@ packages:
         optional: true
     dependencies:
       '@types/eslint-scope': 3.7.7
-      '@types/estree': 1.0.6
+      '@types/estree': 1.0.7
       '@webassemblyjs/ast': 1.14.1
       '@webassemblyjs/wasm-edit': 1.14.1
       '@webassemblyjs/wasm-parser': 1.14.1


Reviews:
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-15T16:15:38Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-17T15:46:28Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-17T16:02:24Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T18:25:17Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T18:26:21Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T18:26:38Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T18:28:07Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-25T08:29:22Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-25T08:38:19Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-25T08:41:21Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-25T08:41:45Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T17:00:51Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T17:12:59Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T17:15:17Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T17:23:40Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T18:49:43Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T18:51:33Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T18:52:00Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T18:52:33Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T18:53:08Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-29T10:20:32Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-29T10:24:19Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-29T14:33:02Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-29T14:34:50Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-29T14:45:52Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-29T14:52:10Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T13:54:20Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T15:13:36Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T15:15:18Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T15:16:09Z
user: login: josephjclark
body: I'd like to see more tests generally. There's a lot of heavy logic in `responseWithPagination` and I don't really think the tests are covering all the cases

Basically, I need to be 100% satisfied that this helper function works. If I'm satsifed, I can quickly review the other tests and be confident that the adaptor is working as designed.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T15:20:37Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T13:38:55Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T13:48:27Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-27T16:13:11Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-27T16:13:33Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-27T16:14:37Z
user: login: josephjclark
body: Hi @mtuchi - this is looking pretty good. I think it's the simplest, clearest implementation yet, which is a really good sign.

I'm sorry but I'm going to push you a bit further on the tests. I think it's super important that the behaviours are correct and the test functions do what they are supposed to do (or what they claim to do).

I know it's painful and taking a lot of time, but it's also a really important learning opportunity.

When writing your tests, try to be really clear in your mind _exactly_ what you want to test, make sure the behaviour/expectation is at least roughly reflected in the test name, and make sure the test code is as simple and focused as it can possibly be.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-28T12:15:31Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-29T16:05:27Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-29T16:09:36Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-29T16:11:01Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-29T16:12:17Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-29T16:13:38Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-29T16:22:35Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-30T06:48:14Z
user: login: mtuchi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-30T06:49:41Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-30T06:53:58Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-30T06:54:51Z
user: login: josephjclark
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-30T06:56:44Z

Review comments:
diff_hunk: @@ -52,52 +44,76 @@ export function execute(...operations) {
 export function getForms() {
   return async state => {
     const url = `/assets/?asset_type=survey`;
-
     const response = await util.request(state, 'GET', url, {});
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
+    console.log('✓', response.body.results?.length, 'forms fetched.');
+
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', {limit: Infinity});
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Optional query params for the request
+ * @param {number} [options.limit] - Limit the number of submissions to fetch
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: What do you think about adding something like: `restricts query to a single request. Use max to allow multiple requests` 
created_at: 2025-04-15T16:15:38Z
author_association: COLLABORATOR
diff_hunk: @@ -52,52 +44,76 @@ export function execute(...operations) {
 export function getForms() {
   return async state => {
     const url = `/assets/?asset_type=survey`;
-
     const response = await util.request(state, 'GET', url, {});
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
+    console.log('✓', response.body.results?.length, 'forms fetched.');
+
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', {limit: Infinity});
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Optional query params for the request
+ * @param {number} [options.limit] - Limit the number of submissions to fetch
path: packages/kobotoolbox/src/Adaptor.js
user: login: mtuchi
body: I don't think this is true, Because it depends on `pageSize`
created_at: 2025-04-17T15:46:28Z
author_association: COLLABORATOR
diff_hunk: @@ -52,52 +44,76 @@ export function execute(...operations) {
 export function getForms() {
   return async state => {
     const url = `/assets/?asset_type=survey`;
-
     const response = await util.request(state, 'GET', url, {});
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
+    console.log('✓', response.body.results?.length, 'forms fetched.');
+
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', {limit: Infinity});
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Optional query params for the request
+ * @param {number} [options.limit] - Limit the number of submissions to fetch
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: If limit is set, pageSize and max are ignored
```
if (keysToRemove.length) {
        console.warn(
          `Warning: ignoring option [${keysToRemove.join(
            ','
          )}] as "limit" is set`
        );
        delete resolvedOptions.max;
        delete resolvedOptions.pageSize;
      }
```
created_at: 2025-04-17T16:02:24Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,122 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const { data = {}, query = {}, headers = {}, parseAs = 'json' } = opts;
+
+  const requestPath = cleanPath(`/api/${apiVersion}/${path}`);
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: This needs to pass the query object through, like:
```
return commonRequest(method, requestPath, options).then(res =>
    logResponse(res, query)
  );
```

That we we get the query logged in the console.

We  fixed this in common recently but I fear we may have gotten it wrong, see #1163
created_at: 2025-04-24T18:25:17Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,122 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const { data = {}, query = {}, headers = {}, parseAs = 'json' } = opts;
+
+  const requestPath = cleanPath(`/api/${apiVersion}/${path}`);
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
+}
+
+export async function requestWithPagination(state, path, options = {}) {
+  const results = [];
+
+  let { pageSize, start, limit, max, ...otherOptions } = options;
+
+  const isUsingDefaultMax = max === undefined;
+  const maxResults = max ?? limit ?? 1e4;
+
+  let isFirstRequest = true;
+  let requestOptions = { query: { start, limit }, ...otherOptions };
+  let shouldFetchMoreContent = false;
+  const didUserPassLimit = Boolean(max || limit);
+
+  do {
+    requestOptions.query ??= {};
+
+    if (!isNaN(start)) {
+      requestOptions.query.start = start;
+    }
+
+    if (limit) {
+      requestOptions.query.limit = limit;
+    } else if (didUserPassLimit || !isFirstRequest) {
+      // If there's an explicit limit or page size,
+      // or this is not the first request,
+      // set the limit in the URL
+      requestOptions.query.limit = Math.min(
+        pageSize || maxResults,
+        maxResults - results.length
+      );
+    } else if (!isNaN(pageSize)) {
+      requestOptions.query.limit = pageSize;
+    }
+
+    const response = await request(state, 'GET', path, requestOptions);
+
+    if (response.body.results) {
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: This needs to be `response.body?.results`, in case there's no response body (I'm seeing this right now with a 301 response)

Or maybe if there's no body we exit the loop?
created_at: 2025-04-24T18:26:20Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,122 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const { data = {}, query = {}, headers = {}, parseAs = 'json' } = opts;
+
+  const requestPath = cleanPath(`/api/${apiVersion}/${path}`);
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
+}
+
+export async function requestWithPagination(state, path, options = {}) {
+  const results = [];
+
+  let { pageSize, start, limit, max, ...otherOptions } = options;
+
+  const isUsingDefaultMax = max === undefined;
+  const maxResults = max ?? limit ?? 1e4;
+
+  let isFirstRequest = true;
+  let requestOptions = { query: { start, limit }, ...otherOptions };
+  let shouldFetchMoreContent = false;
+  const didUserPassLimit = Boolean(max || limit);
+
+  do {
+    requestOptions.query ??= {};
+
+    if (!isNaN(start)) {
+      requestOptions.query.start = start;
+    }
+
+    if (limit) {
+      requestOptions.query.limit = limit;
+    } else if (didUserPassLimit || !isFirstRequest) {
+      // If there's an explicit limit or page size,
+      // or this is not the first request,
+      // set the limit in the URL
+      requestOptions.query.limit = Math.min(
+        pageSize || maxResults,
+        maxResults - results.length
+      );
+    } else if (!isNaN(pageSize)) {
+      requestOptions.query.limit = pageSize;
+    }
+
+    const response = await request(state, 'GET', path, requestOptions);
+
+    if (response.body.results) {
+      results.push(...response.body.results);
+
+      if (!start) {
+        start = 1;
+      }
+      if (response?.body?.next) {
+        const nextUrl = new URL(response?.body?.next);
+        start = nextUrl.searchParams.get('start');
+      }
+    } else {
+      results.push(response.body);
+    }
+
+    if (isFirstRequest)
+      if (!pageSize) {
+        pageSize = results.length;
+      }
+
+    isFirstRequest = false;
+
+    const hasMoreContent = response.body.next?.includes('start=')
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: This also needs to be `response.body?.next?`
created_at: 2025-04-24T18:26:37Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,27 @@
+---
path: .changeset/tame-eggs-notice.md
user: login: josephjclark
body: 
I've made some tweaks (but this may change again)
````
---
'@openfn/language-kobotoolbox': major
---

Improved pagination support in `getSubmissions()`

- Removed `paginate` option
- Added new pagination options:
  - `max`: Maximum number of results to fetch (paginated fetch)
  - `pageSize`: Number of items per page in a paginated request
  - `limit`: Force a single request of a fixed size (disables pagination)

### Examples

```js
// Fetch submissions with default limit (10000)
getSubmissions('form-id');

// Fetch all submissions, with pagination
getSubmissions('form-id', { max: Infinity });

// Fetch no more than 500 submissions, with pagination
getSubmissions('form-id', { max: 500 });

// Fetch 100 submissions in a single request
getSubmissions('form-id', { limit: 100 });
```

````
created_at: 2025-04-24T18:28:06Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,122 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const { data = {}, query = {}, headers = {}, parseAs = 'json' } = opts;
+
+  const requestPath = cleanPath(`/api/${apiVersion}/${path}`);
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: I think we should NOT fix this here and wait for a fix to common.

Or be really audacious and fix common on a real quick new PR :scream: 
created_at: 2025-04-25T08:29:22Z
author_association: COLLABORATOR
diff_hunk: @@ -52,52 +44,77 @@ export function execute(...operations) {
 export function getForms() {
   return async state => {
     const url = `/assets/?asset_type=survey`;
-
     const response = await util.request(state, 'GET', url, {});
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
+    console.log('✓', response.body.results?.length, 'forms fetched.');
+
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Optional query params for the request
+ * @param {number} [options.limit] - Restricts query to a single request. Use max to allow multiple requests
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: Thank you for these docs!
created_at: 2025-04-25T08:38:19Z
author_association: COLLABORATOR
diff_hunk: @@ -52,52 +44,77 @@ export function execute(...operations) {
 export function getForms() {
   return async state => {
     const url = `/assets/?asset_type=survey`;
-
     const response = await util.request(state, 'GET', url, {});
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
+    console.log('✓', response.body.results?.length, 'forms fetched.');
+
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Optional query params for the request
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: These are NOT URL query parameters, and they are all openfn only
created_at: 2025-04-25T08:41:21Z
author_association: COLLABORATOR
diff_hunk: @@ -52,52 +44,77 @@ export function execute(...operations) {
 export function getForms() {
   return async state => {
     const url = `/assets/?asset_type=survey`;
-
     const response = await util.request(state, 'GET', url, {});
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
+    console.log('✓', response.body.results?.length, 'forms fetched.');
+
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Optional query params for the request
+ * @param {number} [options.limit] - Restricts query to a single request. Use max to allow multiple requests
+ * @param {object} [options.query] - Query parameters to filter the submissions. See query operators {@link http://docs.mongodb.org/manual/reference/operator/query/.}
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: This is a query but not `query parameters`
created_at: 2025-04-25T08:41:45Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,122 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const { data = {}, query = {}, headers = {}, parseAs = 'json' } = opts;
+
+  const requestPath = cleanPath(`/api/${apiVersion}/${path}`);
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
+}
+
+export async function requestWithPagination(state, path, options = {}) {
+  const results = [];
+
+  let { pageSize, start, limit, max, ...otherOptions } = options;
+
+  const isUsingDefaultMax = max === undefined;
+  const maxResults = max ?? limit ?? 1e4;
+
+  let isFirstRequest = true;
+  let requestOptions = { query: { start, limit }, ...otherOptions };
+  let shouldFetchMoreContent = false;
+  const didUserPassLimit = Boolean(max || limit);
+
+  do {
+    requestOptions.query ??= {};
+
+    if (!isNaN(start)) {
+      requestOptions.query.start = start;
+    }
+
+    if (limit) {
+      requestOptions.query.limit = limit;
+    } else if (didUserPassLimit || !isFirstRequest) {
+      // If there's an explicit limit or page size,
+      // or this is not the first request,
+      // set the limit in the URL
+      requestOptions.query.limit = Math.min(
+        pageSize || maxResults,
+        maxResults - results.length
+      );
+    } else if (!isNaN(pageSize)) {
+      requestOptions.query.limit = pageSize;
+    }
+
+    const response = await request(state, 'GET', path, requestOptions);
+
+    if (response.body.results) {
path: packages/kobotoolbox/src/util.js
user: login: mtuchi
body: Fixed
created_at: 2025-04-28T17:00:50Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,27 @@
+---
path: .changeset/tame-eggs-notice.md
user: login: mtuchi
body: Yeah this has changed to include `http` changes 
created_at: 2025-04-28T17:12:59Z
author_association: COLLABORATOR
diff_hunk: @@ -52,52 +44,77 @@ export function execute(...operations) {
 export function getForms() {
   return async state => {
     const url = `/assets/?asset_type=survey`;
-
     const response = await util.request(state, 'GET', url, {});
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
+    console.log('✓', response.body.results?.length, 'forms fetched.');
+
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Optional query params for the request
+ * @param {number} [options.limit] - Restricts query to a single request. Use max to allow multiple requests
+ * @param {object} [options.query] - Query parameters to filter the submissions. See query operators {@link http://docs.mongodb.org/manual/reference/operator/query/.}
path: packages/kobotoolbox/src/Adaptor.js
user: login: mtuchi
body: Updated the docs
created_at: 2025-04-28T17:15:16Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,122 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const { data = {}, query = {}, headers = {}, parseAs = 'json' } = opts;
+
+  const requestPath = cleanPath(`/api/${apiVersion}/${path}`);
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
path: packages/kobotoolbox/src/util.js
user: login: mtuchi
body: Fixed ✅ 
created_at: 2025-04-28T17:23:40Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+---
+'@openfn/language-kobotoolbox': major
+---
+
+Improved pagination support and restricted HTTP methods to KoboToolbox API
+
+### Breaking Changes
+
+- Updated `getSubmission` pagination behavior:
+
+  - Removed `paginate` option
+  - Added new pagination options:
+    - `limit`: Controls number of results returned
+    - `pageSize`: Number of items per page
+
+- Remove export of `http` methods from `@openfn/language-common`
+- Add support for `maxRedirections` option in `HTTPRequestOptions`
+
+### Migration Guide
+
+1. Pagination Changes:
+
+   - Replace `paginate: true/false` with `limit` and `pageSize` options
+   - Default limit is `10000` if not specified
+
+   ```js
+   // Fetch submissions with default limit (10000)
+   getSubmission('form-id');
+
+   // Fetch all submissions (no limit)
+   getSubmission('form-id', { limit: Infinity });
+
+   // Fetch submissions with max results
+   getSubmission('form-id', { limit: 500 }); // Returns up to 500 submissions
+   ```
+
+2. HTTP Changes:
+
+- For KoboToolbox API requests: Continue using the built-in methods
path: .changeset/tame-eggs-notice.md
user: login: josephjclark
body: Was anyone using general purpose requests? I'm a bit suspicious of this.

I consider `http.get('https://any-api.com/endpoint');` to be invalid, undocumented behaviour. We don't tell anyone to do this (right??), and don't document it. So we can just remove this behaviour without making a big deal about it. The http namespace is also very new so not many users should be affected.

By documenting the breaking change like this, we sort of imply that the old behaviour was supported.

I wouldn't even report it as a bug fix or feature change or anything. I'd just say (when we're ready) `http methods will now throw an error when passing absolute URLs`. No other code examples, workarounds or docs are needed.
created_at: 2025-04-28T18:49:43Z
author_association: COLLABORATOR
diff_hunk: @@ -51,53 +43,69 @@ export function execute(...operations) {
  */
 export function getForms() {
   return async state => {
-    const url = `/assets/?asset_type=survey`;
+    const response = await util.request(state, 'GET', 'assets', {
+      query: { asset_type: 'survey' },
+    });
 
-    const response = await util.request(state, 'GET', url, {});
+    console.log('✓', response.body.results?.length, 'forms fetched.');
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: Ok, but can we add a an example like:
```
 * @example <caption>Get submissions for a specific form</caption>
 * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
 *
```

95% of our users do not need to worry about the limit we impose. And if they ever hit it they'll see a nice error right?
created_at: 2025-04-28T18:51:33Z
author_association: COLLABORATOR
diff_hunk: @@ -51,53 +43,69 @@ export function execute(...operations) {
  */
 export function getForms() {
   return async state => {
-    const url = `/assets/?asset_type=survey`;
+    const response = await util.request(state, 'GET', 'assets', {
+      query: { asset_type: 'survey' },
+    });
 
-    const response = await util.request(state, 'GET', url, {});
+    console.log('✓', response.body.results?.length, 'forms fetched.');
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Optional query params for the request
+ * @param {object} [options.query] - (Openfn only) Query options to filter the submissions. See query operators {@link http://docs.mongodb.org/manual/reference/operator/query/.}
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: Please remove `openfn only` from these docs. It's super confusing.
created_at: 2025-04-28T18:52:00Z
author_association: COLLABORATOR
diff_hunk: @@ -51,53 +43,69 @@ export function execute(...operations) {
  */
 export function getForms() {
   return async state => {
-    const url = `/assets/?asset_type=survey`;
+    const response = await util.request(state, 'GET', 'assets', {
+      query: { asset_type: 'survey' },
+    });
 
-    const response = await util.request(state, 'GET', url, {});
+    console.log('✓', response.body.results?.length, 'forms fetched.');
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Optional query params for the request
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: These are not "query params". You can just say `Options to control the request`
created_at: 2025-04-28T18:52:33Z
author_association: COLLABORATOR
diff_hunk: @@ -51,53 +43,69 @@ export function execute(...operations) {
  */
 export function getForms() {
   return async state => {
-    const url = `/assets/?asset_type=survey`;
+    const response = await util.request(state, 'GET', 'assets', {
+      query: { asset_type: 'survey' },
+    });
 
-    const response = await util.request(state, 'GET', url, {});
+    console.log('✓', response.body.results?.length, 'forms fetched.');
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });
  * @example <caption>Get form submissions with a query</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2022-06-12T21:54:20" } } });
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { query: { _submission_time:{ $gte: "2025-03-12T21:54:20" } } });
  * @function
  * @public
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Optional query params for the request
+ * @param {object} [options.query] - (Openfn only) Query options to filter the submissions. See query operators {@link http://docs.mongodb.org/manual/reference/operator/query/.}
+ * @param {number} [options.limit=10000] - (Openfn only) Maximum number of submissions to fetch.
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: Here would be a good place to say `Pass Infinity to disable the limit and download all submissions"
created_at: 2025-04-28T18:53:08Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+---
+'@openfn/language-kobotoolbox': major
+---
+
+Improved pagination support and restricted HTTP methods to KoboToolbox API
+
+### Breaking Changes
+
+- Updated `getSubmission` pagination behavior:
+
+  - Removed `paginate` option
+  - Added new pagination options:
+    - `limit`: Controls number of results returned
+    - `pageSize`: Number of items per page
+
+- Remove export of `http` methods from `@openfn/language-common`
+- Add support for `maxRedirections` option in `HTTPRequestOptions`
+
+### Migration Guide
+
+1. Pagination Changes:
+
+   - Replace `paginate: true/false` with `limit` and `pageSize` options
+   - Default limit is `10000` if not specified
+
+   ```js
+   // Fetch submissions with default limit (10000)
+   getSubmission('form-id');
+
+   // Fetch all submissions (no limit)
+   getSubmission('form-id', { limit: Infinity });
+
+   // Fetch submissions with max results
+   getSubmission('form-id', { limit: 500 }); // Returns up to 500 submissions
+   ```
+
+2. HTTP Changes:
+
+- For KoboToolbox API requests: Continue using the built-in methods
path: .changeset/tame-eggs-notice.md
user: login: mtuchi
body: In v2 we don't have any job that use `http.get()` to access a different rest api. 
Regarding removing `http` export from common from v1.0.4 to v2.4.3 `http` method was exported from common.
So a request like this would work 
```js
http.get({ url: 'https://jsonplaceholder.typicode.com/todos' })
```

In v3.0.0 we introduces kobotoolbox `http` methods. So this means we can't make request outside kobotoolbox
```js
http.get('https://any-api.com/endpoint');
```
But the `http` common export was not removed.

In this PR, i have removed the common export of http from common

I have removed the example in changeset and left the one line description for http changes
created_at: 2025-04-29T10:20:32Z
author_association: COLLABORATOR
diff_hunk: @@ -51,53 +43,69 @@ export function execute(...operations) {
  */
 export function getForms() {
   return async state => {
-    const url = `/assets/?asset_type=survey`;
+    const response = await util.request(state, 'GET', 'assets', {
+      query: { asset_type: 'survey' },
+    });
 
-    const response = await util.request(state, 'GET', url, {});
+    console.log('✓', response.body.results?.length, 'forms fetched.');
 
-    console.log('✓', response.body.results.length, 'forms fetched.');
     return util.prepareNextState(state, response);
   };
 }
 
 /**
  * Get submissions for a specific form. Calls `/api/v2/assets/<formId>/data/`.
  * @example <caption>Get all submissions for a specific form</caption>
- * getSubmissions('aXecHjmbATuF6iGFmvBLBX');
+ * getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit: Infinity });
path: packages/kobotoolbox/src/Adaptor.js
user: login: mtuchi
body: Example added ✅ ,

Regarding the limit we impose, There is a warning message that will appear if we have more items to download but have reached the maximum results limit
created_at: 2025-04-29T10:24:18Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,126 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export async function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const {
+    data = {},
+    query = {},
+    headers = {},
+    parseAs = 'json',
+    maxRedirections,
+  } = opts;
+
+  const requestPath = `/api/${apiVersion}/${path}/`;
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    maxRedirections,
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(res => {
+    logResponse(res, options.query);
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: I think we agreed to not pass options.query here, right? We need to fix this in common #1163 
created_at: 2025-04-29T14:33:02Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,242 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const responseWithPagination = (url, objects, query) => {
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: Didn't we agree to move `responseWithpagination` into its own file, write a set of unit tests against it, and re-use it in both `http.test.js` and `common.test.js`?

Quite important to get this done before we merge I think
created_at: 2025-04-29T14:34:50Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,126 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export async function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const {
+    data = {},
+    query = {},
+    headers = {},
+    parseAs = 'json',
+    maxRedirections,
+  } = opts;
+
+  const requestPath = `/api/${apiVersion}/${path}/`;
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    maxRedirections,
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(res => {
+    logResponse(res, options.query);
path: packages/kobotoolbox/src/util.js
user: login: mtuchi
body: My bad 🙌🏽 , i might have missed this during changes. Let me revert back
created_at: 2025-04-29T14:45:52Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,242 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const responseWithPagination = (url, objects, query) => {
path: packages/kobotoolbox/test/adaptor.test.js
user: login: mtuchi
body: You're right, I had overlooked this. Appreciate the reminder — I’ll get on it!
created_at: 2025-04-29T14:52:09Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,242 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const responseWithPagination = (url, objects, query) => {
path: packages/kobotoolbox/test/adaptor.test.js
user: login: mtuchi
body: Hiya @josephjclark  👋🏽 , I have created `responseWithPagination` helper for unit test and i have reused the helper in `adaptor.test.js` and `util.test.js`
created_at: 2025-05-01T13:54:19Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,47 @@
+export const responseWithPagination = (url, objects, options) => {
+  const { pageSize, start, limit } = options;
path: packages/kobotoolbox/test/helper.js
user: login: josephjclark
body: This shouldn't have pagesize. The helper only ever returns a single page of data, right? So it just needs to take start and limit.

Having a pagesize here just adds confusion and means I can't entirely trust the tests
created_at: 2025-05-01T15:13:36Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,47 @@
+export const responseWithPagination = (url, objects, options) => {
+  const { pageSize, start, limit } = options;
path: packages/kobotoolbox/test/helper.js
user: login: josephjclark
body: Or am I misunderstanding what this helper does?
created_at: 2025-05-01T15:15:18Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,221 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+  it('should return the first item', () => {
+    const start = 0;
+    const limit = 1;
+    const { results } = responseWithPagination('www', items, {
+      limit,
+      start,
+    });
+    expect(results).to.eql([1]);
+  });
+  it('should return 2 items with limit 2', () => {
+    const start = 0;
+    const limit = 2;
+    const { results } = responseWithPagination('www', items, {
+      limit,
+      start,
+    });
+    expect(results).to.eql([1, 2]);
+  });
+  it('should return 2 items with start 1 and pageSize 2', () => {
+    const start = 1;
+    const pageSize = 2;
+    const { results } = responseWithPagination('www', items, {
+      pageSize,
+      start,
+    });
+    expect(results).to.eql([2, 3]);
+  });
+  it('should return all items', () => {
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: Why? What is this actually testing?
created_at: 2025-05-01T15:16:09Z
author_association: COLLABORATOR
diff_hunk: @@ -25,100 +26,64 @@ const defaultObjects = [
   { uid: '6' },
 ];
 
-const responseWithPagination = (
-  url,
-  pageSize = 1000,
-  start = 0,
-  objects = defaultObjects
-) => {
-  const s = parseInt(start);
-  const l = parseInt(pageSize);
-  const results = objects.slice(s, s + l);
-  const count = objects.length;
-  const previous = `${url}?format=json&start=${s}&limit=${l}`;
-  let next = null;
-  if (count > s + l) {
-    next = `${url}?format=json&start=${s + l}&limit=${l}`;
-
-    return {
-      next,
-      count,
-      previous,
-      results,
-    };
-  } else {
-    return { results, count, previous, next };
-  }
-};
-
 describe('responseWithPagination', () => {
   const items = [1, 2, 3];
   it('should return the first item', () => {
     const start = 0;
-    const pageSize = 1;
-    const { next, results } = responseWithPagination(
-      'www',
-      pageSize,
+    const limit = 1;
+    const { results } = responseWithPagination('www', items, {
+      limit,
       start,
-      items
-    );
+    });
     expect(results).to.eql([1]);
   });
-  it('should return 2 items with pageSize 2', () => {
+  it('should return 2 items with limit 2', () => {
     const start = 0;
-    const pageSize = 2;
-    const { next, results } = responseWithPagination(
-      'www',
-      pageSize,
+    const limit = 2;
+    const { results } = responseWithPagination('www', items, {
+      limit,
       start,
-      items
-    );
+    });
     expect(results).to.eql([1, 2]);
   });
   it('should return 2 items with start 1 and pageSize 2', () => {
     const start = 1;
     const pageSize = 2;
-    const { next, results } = responseWithPagination(
-      'www',
+    const { results } = responseWithPagination('www', items, {
       pageSize,
       start,
-      items
-    );
+    });
     expect(results).to.eql([2, 3]);
   });
   it('should return all items', () => {
     const start = 0;
     const pageSize = 3;
-    const { next, results } = responseWithPagination(
-      'www',
+    const { next, results } = responseWithPagination('www', items, {
       pageSize,
       start,
-      items
-    );
+    });
+
     expect(results).to.eql([1, 2, 3]);
   });
   it('should return all items if pageSize is greater', () => {
     const start = 0;
     const pageSize = 100;
-    const { next, results } = responseWithPagination(
-      'www',
+    const { next, results } = responseWithPagination('www', items, {
       pageSize,
       start,
-      items
-    );
+    });
     expect(results).to.eql([1, 2, 3]);
   });
   it('should return the correct next link', () => {
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: hmm, I think you should assert the `next` link in every test. Even if undefined. Then you need to rethink the purpose of this test.

because it's hard to test the url string, you might prefer to return like a `_next` object with all the key/value pairs. That's not used by the adaptor, but it can be used in these tests to ensure we're getting the correct behaviour

created_at: 2025-05-01T15:19:24Z
author_association: COLLABORATOR
diff_hunk: @@ -66,8 +66,8 @@ export function getForms() {
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Options to control the request
  * @param {object} [options.query] - Query options to filter the submissions. See query operators {@link http://docs.mongodb.org/manual/reference/operator/query/.}
- * @param {number} [options.limit=10000] - Maximum number of submissions to fetch. Pass Infinity to disable the limit and download all submissions
- * @param {number} [options.pageSize=1000] - Limits the size of each page of submissions. Maximum value is 30000.
+ * @param {number} [options.limit=30000] - Maximum number of submissions to fetch. Pass Infinity to disable the limit and download all submissions
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: This change is a surprise?
created_at: 2025-05-15T09:24:03Z
author_association: COLLABORATOR
diff_hunk: @@ -66,8 +66,8 @@ export function getForms() {
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Options to control the request
  * @param {object} [options.query] - Query options to filter the submissions. See query operators {@link http://docs.mongodb.org/manual/reference/operator/query/.}
- * @param {number} [options.limit=10000] - Maximum number of submissions to fetch. Pass Infinity to disable the limit and download all submissions
- * @param {number} [options.pageSize=1000] - Limits the size of each page of submissions. Maximum value is 30000.
+ * @param {number} [options.limit=30000] - Maximum number of submissions to fetch. Pass Infinity to disable the limit and download all submissions
path: packages/kobotoolbox/src/Adaptor.js
user: login: josephjclark
body: I think `limit` - the absolute maximum number of items returned by our adaptor, should be 10k. It could be 5k or 20k or 13.5k but even 30k. It's arbitrary. This NOT trying to emulate the kobo limit or anything.

10k feels like a good number and I don't understand why it changed.

Similarly I don't know why `pageSize` changed. We could default page size in the adaptor (so we always send `limit=1000`), or we could leave it at the kobo default, which would be 30k. Which is fine, but we should't have a default page size that's bigger than the default limit - that makes no sense. And since w'ere controlling page sizes, having smaller pages means we're less likely to have requests timeout or fail for any reason.
created_at: 2025-05-15T13:48:27Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,47 @@
+export const responseWithPagination = (url, objects, options) => {
+  const { pageSize, start, limit } = options;
path: packages/kobotoolbox/test/helper.js
user: login: mtuchi
body: I have updated the helper function, please have another look
created_at: 2025-05-27T16:13:11Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,221 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+  it('should return the first item', () => {
+    const start = 0;
+    const limit = 1;
+    const { results } = responseWithPagination('www', items, {
+      limit,
+      start,
+    });
+    expect(results).to.eql([1]);
+  });
+  it('should return 2 items with limit 2', () => {
+    const start = 0;
+    const limit = 2;
+    const { results } = responseWithPagination('www', items, {
+      limit,
+      start,
+    });
+    expect(results).to.eql([1, 2]);
+  });
+  it('should return 2 items with start 1 and pageSize 2', () => {
+    const start = 1;
+    const pageSize = 2;
+    const { results } = responseWithPagination('www', items, {
+      pageSize,
+      start,
+    });
+    expect(results).to.eql([2, 3]);
+  });
+  it('should return all items', () => {
path: packages/kobotoolbox/test/util.test.js
user: login: mtuchi
body: I have updated the tests
created_at: 2025-05-27T16:13:33Z
author_association: COLLABORATOR
diff_hunk: @@ -66,8 +66,8 @@ export function getForms() {
  * @param {string} formId - Form Id to get the specific submissions
  * @param {object} [options={}] - Options to control the request
  * @param {object} [options.query] - Query options to filter the submissions. See query operators {@link http://docs.mongodb.org/manual/reference/operator/query/.}
- * @param {number} [options.limit=10000] - Maximum number of submissions to fetch. Pass Infinity to disable the limit and download all submissions
- * @param {number} [options.pageSize=1000] - Limits the size of each page of submissions. Maximum value is 30000.
+ * @param {number} [options.limit=30000] - Maximum number of submissions to fetch. Pass Infinity to disable the limit and download all submissions
path: packages/kobotoolbox/src/Adaptor.js
user: login: mtuchi
body: I have updated the docs and add more unit tests
created_at: 2025-05-27T16:14:36Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: What is this actually testing?

I think this probably needs to be called `Use the default limit if no limit is passed`

And then to test that, you need to pass `{ defaultLimit: 1 }` and assert that only a single item comes back.

Then you need another test like `Use the user limit, not the default limit`, passing `limit: 2` and `defaultLimit:1`, and ensure that 2 items come back.

I don't think I'd include pagination url/next etc in this same test. I'd have a separate test like "if there are more items available, pagination values are set on the response"

I don't think I'd bother to test the default start - but I would test that if `start: 1`, the second item gets returned
created_at: 2025-05-28T10:50:36Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
+    const { results, next, previous, requestCount } = responseWithPagination(
+      items,
+      {},
+      { url: 'www' }
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: Do not pass `url` here because it doesn't matter. It confuses the test. This might be true of all tests?

You might want to test that if url is set, and there are more items, the url is used to construct the `next` string
created_at: 2025-05-28T10:50:59Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
+    const { results, next, previous, requestCount } = responseWithPagination(
+      items,
+      {},
+      { url: 'www' }
+    );
+
+    expect(next).to.eql(null);
+    expect(previous).to.eql(null);
+    expect(results).to.eql(items);
+    expect(requestCount).to.eql(1);
+    expect(results.length).to.eql(3);
+  });
+  it('limit is greater than maxLimit', () => {
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: This test name needs to be something like: `Don't return more items than maximum page size`

The test name here doesn't make it clear what is being tested here, nor what the correct behaviour is.
created_at: 2025-05-28T10:53:39Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
+    const { results, next, previous, requestCount } = responseWithPagination(
+      items,
+      {},
+      { url: 'www' }
+    );
+
+    expect(next).to.eql(null);
+    expect(previous).to.eql(null);
+    expect(results).to.eql(items);
+    expect(requestCount).to.eql(1);
+    expect(results.length).to.eql(3);
+  });
+  it('limit is greater than maxLimit', () => {
+    const start = 0;
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: Don't pass start in this test, it's not  relevant
created_at: 2025-05-28T10:53:55Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
+    const { results, next, previous, requestCount } = responseWithPagination(
+      items,
+      {},
+      { url: 'www' }
+    );
+
+    expect(next).to.eql(null);
+    expect(previous).to.eql(null);
+    expect(results).to.eql(items);
+    expect(requestCount).to.eql(1);
+    expect(results.length).to.eql(3);
+  });
+  it('limit is greater than maxLimit', () => {
+    const start = 0;
+    const limit = 4e3;
+    const { results, next, previous } = responseWithPagination(
+      items,
+      {
+        limit,
+        start,
+      },
+      { url: 'www' }
+    );
+
+    expect(results).to.eql(items);
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: The problem with this test, which is really important, is that there are only 3 items. The test will still pass if limit is less than maxLimit.

You either need to increase the number of items (if the maxLimit is `10k`, then you need to request `10k +1` items, and ensure that the result array length is `10k`) or reduce the limits (set `maxLimit` to 1 and `limit` to 2 - only one item should be returned.
created_at: 2025-05-28T10:58:55Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
+    const { results, next, previous, requestCount } = responseWithPagination(
+      items,
+      {},
+      { url: 'www' }
+    );
+
+    expect(next).to.eql(null);
+    expect(previous).to.eql(null);
+    expect(results).to.eql(items);
+    expect(requestCount).to.eql(1);
+    expect(results.length).to.eql(3);
+  });
+  it('limit is greater than maxLimit', () => {
+    const start = 0;
+    const limit = 4e3;
+    const { results, next, previous } = responseWithPagination(
+      items,
+      {
+        limit,
+        start,
+      },
+      { url: 'www' }
+    );
+
+    expect(results).to.eql(items);
+    expect(next).to.eql(null);
+    expect(previous).to.eql(null);
+  });
+  it('limit is greater than number of items', () => {
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: Maybe a better test name is `Return all items if limit is greater than the number of item`. What's important here is that the test name sets the expectation of what should happen.
created_at: 2025-05-28T10:59:56Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
+    const { results, next, previous, requestCount } = responseWithPagination(
+      items,
+      {},
+      { url: 'www' }
+    );
+
+    expect(next).to.eql(null);
+    expect(previous).to.eql(null);
+    expect(results).to.eql(items);
+    expect(requestCount).to.eql(1);
+    expect(results.length).to.eql(3);
+  });
+  it('limit is greater than maxLimit', () => {
+    const start = 0;
+    const limit = 4e3;
+    const { results, next, previous } = responseWithPagination(
+      items,
+      {
+        limit,
+        start,
+      },
+      { url: 'www' }
+    );
+
+    expect(results).to.eql(items);
+    expect(next).to.eql(null);
+    expect(previous).to.eql(null);
+  });
+  it('limit is greater than number of items', () => {
+    const start = 0;
+    const limit = 1000;
+    const { results, next, previous } = responseWithPagination(
+      items,
+      {
+        limit,
+        start,
+      },
+      { url: 'www' }
+    );
+
+    expect(results).to.eql(items);
+    expect(next).to.eql(null);
+    expect(previous).to.eql(null);
+  });
+  it('limit is less than number of items', () => {
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: I'd rename this to `Only return 1 item if the limit is 1`
created_at: 2025-05-28T11:00:59Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,429 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_REQUEST_LIMIT, DEFAULT_MAX_LIMIT } from '../src/util';
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const mockSubmissions = (formId, submissions, opts = {}) => {
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: It looks like this is never called, so this mock is never set up? Is that right?
created_at: 2025-05-28T11:20:22Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,429 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_REQUEST_LIMIT, DEFAULT_MAX_LIMIT } from '../src/util';
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const mockSubmissions = (formId, submissions, opts = {}) => {
+  const { pageSize, ...otherOpts } = opts;
+
+  const path = `/api/v2/assets/${formId}/data/`;
+  const url = `${configuration.baseUrl}${path}`;
+  if (otherOpts.query && typeof otherOpts.query !== 'string') {
+    otherOpts.query = JSON.stringify(otherOpts.query);
+  }
+  testServer
+    .intercept({
+      path,
+      method: 'GET',
+      query: {
+        format: 'json',
+        ...otherOpts,
+      },
+    })
+    .reply(
+      200,
+      (req, res) => {
+        const { query, origin } = req;
+        const results = responseWithPagination(
+          submissions,
+          {
+            limit: otherOpts.limit,
+            start: otherOpts.start,
+          },
+          {
+            url,
+            defaulLimit: pageSize,
+          }
+        );
+        return results;
+      },
+      {
+        ...jsonHeaders,
+      }
+    );
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleResponse = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+  const formId = 'aXecHjmbATuF6iGFmvBLBX';
+
+  it('should return submissions within default page limits', async () => {
+    let requestCount = 0;
+    const totalRequests = Math.ceil(
+      sampleResponse.length / DEFAULT_REQUEST_LIMIT
+    );
+    console.log({ totalRequests });
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: This logging needs removing
created_at: 2025-05-28T11:21:45Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,429 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_REQUEST_LIMIT, DEFAULT_MAX_LIMIT } from '../src/util';
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const mockSubmissions = (formId, submissions, opts = {}) => {
+  const { pageSize, ...otherOpts } = opts;
+
+  const path = `/api/v2/assets/${formId}/data/`;
+  const url = `${configuration.baseUrl}${path}`;
+  if (otherOpts.query && typeof otherOpts.query !== 'string') {
+    otherOpts.query = JSON.stringify(otherOpts.query);
+  }
+  testServer
+    .intercept({
+      path,
+      method: 'GET',
+      query: {
+        format: 'json',
+        ...otherOpts,
+      },
+    })
+    .reply(
+      200,
+      (req, res) => {
+        const { query, origin } = req;
+        const results = responseWithPagination(
+          submissions,
+          {
+            limit: otherOpts.limit,
+            start: otherOpts.start,
+          },
+          {
+            url,
+            defaulLimit: pageSize,
+          }
+        );
+        return results;
+      },
+      {
+        ...jsonHeaders,
+      }
+    );
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleResponse = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+  const formId = 'aXecHjmbATuF6iGFmvBLBX';
+
+  it('should return submissions within default page limits', async () => {
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: The problem with this test is that you've set up all this elaborate stuff - the paginator, the interceptor, the request count, etc.

But there are only two submissions :joy: So only one request will be made and basically pagination won't be tested at all.

If there's a deeper test of pagination later, that's fine, but you should probably simplify this test code.
created_at: 2025-05-28T11:24:21Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,429 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_REQUEST_LIMIT, DEFAULT_MAX_LIMIT } from '../src/util';
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const mockSubmissions = (formId, submissions, opts = {}) => {
+  const { pageSize, ...otherOpts } = opts;
+
+  const path = `/api/v2/assets/${formId}/data/`;
+  const url = `${configuration.baseUrl}${path}`;
+  if (otherOpts.query && typeof otherOpts.query !== 'string') {
+    otherOpts.query = JSON.stringify(otherOpts.query);
+  }
+  testServer
+    .intercept({
+      path,
+      method: 'GET',
+      query: {
+        format: 'json',
+        ...otherOpts,
+      },
+    })
+    .reply(
+      200,
+      (req, res) => {
+        const { query, origin } = req;
+        const results = responseWithPagination(
+          submissions,
+          {
+            limit: otherOpts.limit,
+            start: otherOpts.start,
+          },
+          {
+            url,
+            defaulLimit: pageSize,
+          }
+        );
+        return results;
+      },
+      {
+        ...jsonHeaders,
+      }
+    );
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleResponse = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+  const formId = 'aXecHjmbATuF6iGFmvBLBX';
+
+  it('should return submissions within default page limits', async () => {
+    let requestCount = 0;
+    const totalRequests = Math.ceil(
+      sampleResponse.length / DEFAULT_REQUEST_LIMIT
+    );
+    console.log({ totalRequests });
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleResponse,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+    const { data } = await execute(getSubmissions('aXecHjmbATuF6iGFmvBLBX'))(
+      state
+    );
+    expect(data).to.eql(sampleResponse);
+    expect(requestCount).to.eql(totalRequests);
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: For me reviewing the tests, it's quite hard to know if the request  count is important, or what it is. By using a dynamic count like this, you've hidden some of the important test details away from the reader.

If we're trying to test that getSubmissions will download multiple pages of requests (we DO need to test that somewhere, but I don't know if this is a good place (the test name isn't terribly clear)), it's probably important to set a fixed value here.

Like if I can see `expect(requestCount).to.eql(1);`, it's clear to me that there's only a single request here. Cool. Now I can  go off and look for where multiple requests are tested.
created_at: 2025-05-28T11:26:53Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,429 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_REQUEST_LIMIT, DEFAULT_MAX_LIMIT } from '../src/util';
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const mockSubmissions = (formId, submissions, opts = {}) => {
+  const { pageSize, ...otherOpts } = opts;
+
+  const path = `/api/v2/assets/${formId}/data/`;
+  const url = `${configuration.baseUrl}${path}`;
+  if (otherOpts.query && typeof otherOpts.query !== 'string') {
+    otherOpts.query = JSON.stringify(otherOpts.query);
+  }
+  testServer
+    .intercept({
+      path,
+      method: 'GET',
+      query: {
+        format: 'json',
+        ...otherOpts,
+      },
+    })
+    .reply(
+      200,
+      (req, res) => {
+        const { query, origin } = req;
+        const results = responseWithPagination(
+          submissions,
+          {
+            limit: otherOpts.limit,
+            start: otherOpts.start,
+          },
+          {
+            url,
+            defaulLimit: pageSize,
+          }
+        );
+        return results;
+      },
+      {
+        ...jsonHeaders,
+      }
+    );
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleResponse = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+  const formId = 'aXecHjmbATuF6iGFmvBLBX';
+
+  it('should return submissions within default page limits', async () => {
+    let requestCount = 0;
+    const totalRequests = Math.ceil(
+      sampleResponse.length / DEFAULT_REQUEST_LIMIT
+    );
+    console.log({ totalRequests });
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleResponse,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+    const { data } = await execute(getSubmissions('aXecHjmbATuF6iGFmvBLBX'))(
+      state
+    );
+    expect(data).to.eql(sampleResponse);
+    expect(requestCount).to.eql(totalRequests);
+    expect(data.length).to.eql(sampleResponse.length);
+  });
+
+  it('should return submissions within specified limit', async () => {
+    const limit = 1;
+    const response = sampleResponse.slice(0, limit);
+    const totalRequests = Math.ceil(response.length / DEFAULT_REQUEST_LIMIT);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: Again, all this elaborate pagination is pointless for a test where we literally only request 1 item.
created_at: 2025-05-28T11:27:25Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,429 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_REQUEST_LIMIT, DEFAULT_MAX_LIMIT } from '../src/util';
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const mockSubmissions = (formId, submissions, opts = {}) => {
+  const { pageSize, ...otherOpts } = opts;
+
+  const path = `/api/v2/assets/${formId}/data/`;
+  const url = `${configuration.baseUrl}${path}`;
+  if (otherOpts.query && typeof otherOpts.query !== 'string') {
+    otherOpts.query = JSON.stringify(otherOpts.query);
+  }
+  testServer
+    .intercept({
+      path,
+      method: 'GET',
+      query: {
+        format: 'json',
+        ...otherOpts,
+      },
+    })
+    .reply(
+      200,
+      (req, res) => {
+        const { query, origin } = req;
+        const results = responseWithPagination(
+          submissions,
+          {
+            limit: otherOpts.limit,
+            start: otherOpts.start,
+          },
+          {
+            url,
+            defaulLimit: pageSize,
+          }
+        );
+        return results;
+      },
+      {
+        ...jsonHeaders,
+      }
+    );
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleResponse = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+  const formId = 'aXecHjmbATuF6iGFmvBLBX';
+
+  it('should return submissions within default page limits', async () => {
+    let requestCount = 0;
+    const totalRequests = Math.ceil(
+      sampleResponse.length / DEFAULT_REQUEST_LIMIT
+    );
+    console.log({ totalRequests });
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleResponse,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+    const { data } = await execute(getSubmissions('aXecHjmbATuF6iGFmvBLBX'))(
+      state
+    );
+    expect(data).to.eql(sampleResponse);
+    expect(requestCount).to.eql(totalRequests);
+    expect(data.length).to.eql(sampleResponse.length);
+  });
+
+  it('should return submissions within specified limit', async () => {
+    const limit = 1;
+    const response = sampleResponse.slice(0, limit);
+    const totalRequests = Math.ceil(response.length / DEFAULT_REQUEST_LIMIT);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleResponse,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+
+    const { data } = await execute(
+      getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit })
+    )(state);
+
+    expect(data).to.eql(response);
+    expect(requestCount).to.eql(totalRequests);
+    expect(data.length).to.eql(response.length);
+  });
+
+  it('should not return more items than the default limit', async () => {
+    const manyObjects = Array.from(
+      { length: DEFAULT_MAX_LIMIT + 1 },
+      (_, i) => ({
+        uid: String(i),
+      })
+    );
+    const response = manyObjects.slice(0, DEFAULT_MAX_LIMIT);
+    const totalRequests = Math.ceil(response.length / DEFAULT_REQUEST_LIMIT);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            manyObjects,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX')(state);
+
+    console.log({ requestCount, totalRequests });
+    expect(data.length).to.eql(DEFAULT_MAX_LIMIT);
+    expect(requestCount).to.eql(totalRequests);
+  });
+
+  it('should be able to explicity get more items than the default max limit', async () => {
+    const manyObjects = Array.from(
+      { length: DEFAULT_MAX_LIMIT + 2 },
+      (_, i) => ({
+        uid: String(i),
+      })
+    );
+
+    const totalRequests = Math.ceil(manyObjects.length / DEFAULT_REQUEST_LIMIT);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            manyObjects,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX', {
+      limit: DEFAULT_MAX_LIMIT + 2,
+    })(state);
+    expect(requestCount).to.eql(totalRequests);
+    expect(data.length).to.eql(manyObjects.length);
+  });
+  it('should allow page size to be overridden', async () => {
+    const state = { configuration };
+    const mockResponse = Array.from({ length: 10 }, (_, i) => ({
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: This needs to be called mockData or something, not mockResponse
created_at: 2025-05-28T11:29:13Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,429 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_REQUEST_LIMIT, DEFAULT_MAX_LIMIT } from '../src/util';
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const mockSubmissions = (formId, submissions, opts = {}) => {
+  const { pageSize, ...otherOpts } = opts;
+
+  const path = `/api/v2/assets/${formId}/data/`;
+  const url = `${configuration.baseUrl}${path}`;
+  if (otherOpts.query && typeof otherOpts.query !== 'string') {
+    otherOpts.query = JSON.stringify(otherOpts.query);
+  }
+  testServer
+    .intercept({
+      path,
+      method: 'GET',
+      query: {
+        format: 'json',
+        ...otherOpts,
+      },
+    })
+    .reply(
+      200,
+      (req, res) => {
+        const { query, origin } = req;
+        const results = responseWithPagination(
+          submissions,
+          {
+            limit: otherOpts.limit,
+            start: otherOpts.start,
+          },
+          {
+            url,
+            defaulLimit: pageSize,
+          }
+        );
+        return results;
+      },
+      {
+        ...jsonHeaders,
+      }
+    );
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleResponse = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+  const formId = 'aXecHjmbATuF6iGFmvBLBX';
+
+  it('should return submissions within default page limits', async () => {
+    let requestCount = 0;
+    const totalRequests = Math.ceil(
+      sampleResponse.length / DEFAULT_REQUEST_LIMIT
+    );
+    console.log({ totalRequests });
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleResponse,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+    const { data } = await execute(getSubmissions('aXecHjmbATuF6iGFmvBLBX'))(
+      state
+    );
+    expect(data).to.eql(sampleResponse);
+    expect(requestCount).to.eql(totalRequests);
+    expect(data.length).to.eql(sampleResponse.length);
+  });
+
+  it('should return submissions within specified limit', async () => {
+    const limit = 1;
+    const response = sampleResponse.slice(0, limit);
+    const totalRequests = Math.ceil(response.length / DEFAULT_REQUEST_LIMIT);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleResponse,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+
+    const { data } = await execute(
+      getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit })
+    )(state);
+
+    expect(data).to.eql(response);
+    expect(requestCount).to.eql(totalRequests);
+    expect(data.length).to.eql(response.length);
+  });
+
+  it('should not return more items than the default limit', async () => {
+    const manyObjects = Array.from(
+      { length: DEFAULT_MAX_LIMIT + 1 },
+      (_, i) => ({
+        uid: String(i),
+      })
+    );
+    const response = manyObjects.slice(0, DEFAULT_MAX_LIMIT);
+    const totalRequests = Math.ceil(response.length / DEFAULT_REQUEST_LIMIT);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            manyObjects,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX')(state);
+
+    console.log({ requestCount, totalRequests });
+    expect(data.length).to.eql(DEFAULT_MAX_LIMIT);
+    expect(requestCount).to.eql(totalRequests);
+  });
+
+  it('should be able to explicity get more items than the default max limit', async () => {
+    const manyObjects = Array.from(
+      { length: DEFAULT_MAX_LIMIT + 2 },
+      (_, i) => ({
+        uid: String(i),
+      })
+    );
+
+    const totalRequests = Math.ceil(manyObjects.length / DEFAULT_REQUEST_LIMIT);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            manyObjects,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX', {
+      limit: DEFAULT_MAX_LIMIT + 2,
+    })(state);
+    expect(requestCount).to.eql(totalRequests);
+    expect(data.length).to.eql(manyObjects.length);
+  });
+  it('should allow page size to be overridden', async () => {
+    const state = { configuration };
+    const mockResponse = Array.from({ length: 10 }, (_, i) => ({
+      uid: String(i),
+    }));
+    const pageSize = 1;
+
+    const totalRequests = Math.ceil(mockResponse.length / pageSize);
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: Again I'd like this to be a static value, so that it's super clear how many requests are being made
created_at: 2025-05-28T11:29:43Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,429 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_REQUEST_LIMIT, DEFAULT_MAX_LIMIT } from '../src/util';
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const mockSubmissions = (formId, submissions, opts = {}) => {
+  const { pageSize, ...otherOpts } = opts;
+
+  const path = `/api/v2/assets/${formId}/data/`;
+  const url = `${configuration.baseUrl}${path}`;
+  if (otherOpts.query && typeof otherOpts.query !== 'string') {
+    otherOpts.query = JSON.stringify(otherOpts.query);
+  }
+  testServer
+    .intercept({
+      path,
+      method: 'GET',
+      query: {
+        format: 'json',
+        ...otherOpts,
+      },
+    })
+    .reply(
+      200,
+      (req, res) => {
+        const { query, origin } = req;
+        const results = responseWithPagination(
+          submissions,
+          {
+            limit: otherOpts.limit,
+            start: otherOpts.start,
+          },
+          {
+            url,
+            defaulLimit: pageSize,
+          }
+        );
+        return results;
+      },
+      {
+        ...jsonHeaders,
+      }
+    );
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleResponse = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+  const formId = 'aXecHjmbATuF6iGFmvBLBX';
+
+  it('should return submissions within default page limits', async () => {
+    let requestCount = 0;
+    const totalRequests = Math.ceil(
+      sampleResponse.length / DEFAULT_REQUEST_LIMIT
+    );
+    console.log({ totalRequests });
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleResponse,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+    const { data } = await execute(getSubmissions('aXecHjmbATuF6iGFmvBLBX'))(
+      state
+    );
+    expect(data).to.eql(sampleResponse);
+    expect(requestCount).to.eql(totalRequests);
+    expect(data.length).to.eql(sampleResponse.length);
+  });
+
+  it('should return submissions within specified limit', async () => {
+    const limit = 1;
+    const response = sampleResponse.slice(0, limit);
+    const totalRequests = Math.ceil(response.length / DEFAULT_REQUEST_LIMIT);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleResponse,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+
+    const { data } = await execute(
+      getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit })
+    )(state);
+
+    expect(data).to.eql(response);
+    expect(requestCount).to.eql(totalRequests);
+    expect(data.length).to.eql(response.length);
+  });
+
+  it('should not return more items than the default limit', async () => {
+    const manyObjects = Array.from(
+      { length: DEFAULT_MAX_LIMIT + 1 },
+      (_, i) => ({
+        uid: String(i),
+      })
+    );
+    const response = manyObjects.slice(0, DEFAULT_MAX_LIMIT);
+    const totalRequests = Math.ceil(response.length / DEFAULT_REQUEST_LIMIT);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            manyObjects,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX')(state);
+
+    console.log({ requestCount, totalRequests });
+    expect(data.length).to.eql(DEFAULT_MAX_LIMIT);
+    expect(requestCount).to.eql(totalRequests);
+  });
+
+  it('should be able to explicity get more items than the default max limit', async () => {
+    const manyObjects = Array.from(
+      { length: DEFAULT_MAX_LIMIT + 2 },
+      (_, i) => ({
+        uid: String(i),
+      })
+    );
+
+    const totalRequests = Math.ceil(manyObjects.length / DEFAULT_REQUEST_LIMIT);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            manyObjects,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX', {
+      limit: DEFAULT_MAX_LIMIT + 2,
+    })(state);
+    expect(requestCount).to.eql(totalRequests);
+    expect(data.length).to.eql(manyObjects.length);
+  });
+  it('should allow page size to be overridden', async () => {
+    const state = { configuration };
+    const mockResponse = Array.from({ length: 10 }, (_, i) => ({
+      uid: String(i),
+    }));
+    const pageSize = 1;
+
+    const totalRequests = Math.ceil(mockResponse.length / pageSize);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            mockResponse,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+              defaultLimit: 1,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX', {
+      pageSize,
+    })(state);
+
+    expect(data.length).to.eql(totalRequests);
+    expect(requestCount).to.eql(totalRequests);
+  });
+
+  it('should get ALL items if the item count exceeds the default limit (ie, limit = Infinity)', async () => {
+    const mockResponse = Array.from(
+      { length: DEFAULT_MAX_LIMIT + 1 },
+      (_, i) => ({
+        uid: String(i),
+      })
+    );
+
+    const totalRequests = Math.ceil(
+      mockResponse.length / DEFAULT_REQUEST_LIMIT
+    );
+    console.log({ totalRequests });
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: Logging
created_at: 2025-05-28T11:30:09Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: Having said all that, you have a couple of tests at the end which do a better job of testing the limit.

I think what you should do is review this block of tests, make sure you're happy  with the test names and tat all important things are covered, and then remove any redundant tests. Because I think you have a couple.

I might be able to help you with this over a call (but I don't have much time).

AI might be able to help as well? Don't let it do the work of generating tests for you - ask it to review  for you, identify redundancy, and generate good names for the tests.
created_at: 2025-05-28T11:33:51Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,132 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export const DEFAULT_MAX_LIMIT = 3e4;
+export const DEFAULT_REQUEST_LIMIT = 1e4;
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: Maybe rename to `DEFAULT_PAGE_SIZE`?
created_at: 2025-05-28T12:07:55Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,132 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export const DEFAULT_MAX_LIMIT = 3e4;
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: Maybe rename to `DEFAULT_LIMIT`?

I know it's a little confusing _which_ limit it means. But in practice this is the default limit option for `getSubmissions`, and I think it's clear enough
created_at: 2025-05-28T12:08:34Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,132 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export const DEFAULT_MAX_LIMIT = 3e4;
+export const DEFAULT_REQUEST_LIMIT = 1e4;
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export async function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const {
+    data = {},
+    query = {},
+    headers = {},
+    parseAs = 'json',
+    maxRedirections,
+  } = opts;
+
+  const requestPath = `/api/${apiVersion}/${path}/`;
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    maxRedirections,
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
+}
+
+export async function requestWithPagination(state, path, options = {}) {
+  const results = [];
+
+  let {
+    pageSize = DEFAULT_REQUEST_LIMIT,
+    start,
+    limit,
+    ...otherOptions
+  } = options;
+
+  const isUsingDefaultLimit = limit === undefined;
+  const maxResults = limit ?? DEFAULT_MAX_LIMIT;
+
+  let isFirstRequest = true;
+  let requestOptions = { query: { start, limit: pageSize }, ...otherOptions };
+  let shouldFetchMoreContent = false;
+  const didUserPassLimit = Boolean(limit);
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: Isn't `didUserPassLmit` and `isUsingDefaultLimit` the  same thing?
created_at: 2025-05-28T12:10:38Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,132 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export const DEFAULT_MAX_LIMIT = 3e4;
+export const DEFAULT_REQUEST_LIMIT = 1e4;
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export async function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const {
+    data = {},
+    query = {},
+    headers = {},
+    parseAs = 'json',
+    maxRedirections,
+  } = opts;
+
+  const requestPath = `/api/${apiVersion}/${path}/`;
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    maxRedirections,
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
+}
+
+export async function requestWithPagination(state, path, options = {}) {
+  const results = [];
+
+  let {
+    pageSize = DEFAULT_REQUEST_LIMIT,
+    start,
+    limit,
+    ...otherOptions
+  } = options;
+
+  const isUsingDefaultLimit = limit === undefined;
+  const maxResults = limit ?? DEFAULT_MAX_LIMIT;
+
+  let isFirstRequest = true;
+  let requestOptions = { query: { start, limit: pageSize }, ...otherOptions };
+  let shouldFetchMoreContent = false;
+  const didUserPassLimit = Boolean(limit);
+
+  do {
+    requestOptions.query ??= {};
+
+    if (!isNaN(start)) {
+      requestOptions.query.start = start;
+    }
+
+    if (didUserPassLimit || !isFirstRequest) {
+      // If there's an explicit limit or page size,
+      // or this is not the first request,
+      // set the limit in the URL
+      requestOptions.query.limit = Math.min(
+        pageSize || maxResults,
+        maxResults - results.length
+      );
+    } else if (!isNaN(pageSize)) {
+      requestOptions.query.limit = pageSize;
+    }
+
+    const response = await request(state, 'GET', path, requestOptions);
+
+    if (response.body?.results) {
+      results.push(...response.body.results);
+
+      if (!start) {
+        start = 1;
+      }
+      if (response?.body?.next) {
+        const nextUrl = new URL(response?.body?.next);
+        start = nextUrl.searchParams.get('start');
+      }
+    } else {
+      results.push(response.body);
+    }
+
+    if (isFirstRequest)
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: Should this be:
```
 if (isFirstRequest && !pageSize) {
  pageSize = results.length;
 }
```
created_at: 2025-05-28T12:11:41Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,132 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export const DEFAULT_MAX_LIMIT = 3e4;
+export const DEFAULT_REQUEST_LIMIT = 1e4;
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export async function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const {
+    data = {},
+    query = {},
+    headers = {},
+    parseAs = 'json',
+    maxRedirections,
+  } = opts;
+
+  const requestPath = `/api/${apiVersion}/${path}/`;
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    maxRedirections,
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
+}
+
+export async function requestWithPagination(state, path, options = {}) {
+  const results = [];
+
+  let {
+    pageSize = DEFAULT_REQUEST_LIMIT,
+    start,
+    limit,
+    ...otherOptions
+  } = options;
+
+  const isUsingDefaultLimit = limit === undefined;
+  const maxResults = limit ?? DEFAULT_MAX_LIMIT;
+
+  let isFirstRequest = true;
+  let requestOptions = { query: { start, limit: pageSize }, ...otherOptions };
+  let shouldFetchMoreContent = false;
+  const didUserPassLimit = Boolean(limit);
+
+  do {
+    requestOptions.query ??= {};
+
+    if (!isNaN(start)) {
+      requestOptions.query.start = start;
+    }
+
+    if (didUserPassLimit || !isFirstRequest) {
+      // If there's an explicit limit or page size,
+      // or this is not the first request,
+      // set the limit in the URL
+      requestOptions.query.limit = Math.min(
+        pageSize || maxResults,
+        maxResults - results.length
+      );
+    } else if (!isNaN(pageSize)) {
+      requestOptions.query.limit = pageSize;
+    }
+
+    const response = await request(state, 'GET', path, requestOptions);
+
+    if (response.body?.results) {
+      results.push(...response.body.results);
+
+      if (!start) {
+        start = 1;
+      }
+      if (response?.body?.next) {
+        const nextUrl = new URL(response?.body?.next);
+        start = nextUrl.searchParams.get('start');
+      }
+    } else {
+      results.push(response.body);
+    }
+
+    if (isFirstRequest)
+      if (!pageSize) {
+        pageSize = results.length;
+      }
+
+    isFirstRequest = false;
+
+    const hasMoreContent = response.body.next?.includes('start=')
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: The true/false is totally redundant here - `includes` returns boolean right?
created_at: 2025-05-28T12:12:15Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,132 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export const DEFAULT_MAX_LIMIT = 3e4;
+export const DEFAULT_REQUEST_LIMIT = 1e4;
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export async function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const {
+    data = {},
+    query = {},
+    headers = {},
+    parseAs = 'json',
+    maxRedirections,
+  } = opts;
+
+  const requestPath = `/api/${apiVersion}/${path}/`;
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    maxRedirections,
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
+}
+
+export async function requestWithPagination(state, path, options = {}) {
+  const results = [];
+
+  let {
+    pageSize = DEFAULT_REQUEST_LIMIT,
+    start,
+    limit,
+    ...otherOptions
+  } = options;
+
+  const isUsingDefaultLimit = limit === undefined;
+  const maxResults = limit ?? DEFAULT_MAX_LIMIT;
+
+  let isFirstRequest = true;
+  let requestOptions = { query: { start, limit: pageSize }, ...otherOptions };
+  let shouldFetchMoreContent = false;
+  const didUserPassLimit = Boolean(limit);
+
+  do {
+    requestOptions.query ??= {};
+
+    if (!isNaN(start)) {
+      requestOptions.query.start = start;
+    }
+
+    if (didUserPassLimit || !isFirstRequest) {
+      // If there's an explicit limit or page size,
+      // or this is not the first request,
+      // set the limit in the URL
+      requestOptions.query.limit = Math.min(
+        pageSize || maxResults,
+        maxResults - results.length
+      );
+    } else if (!isNaN(pageSize)) {
+      requestOptions.query.limit = pageSize;
+    }
+
+    const response = await request(state, 'GET', path, requestOptions);
+
+    if (response.body?.results) {
+      results.push(...response.body.results);
+
+      if (!start) {
path: packages/kobotoolbox/src/util.js
user: login: josephjclark
body: Why is this important?

if the user didn't specify start, and we're going to request another page, start will _never_ be 1. I think we should just trust the url that kobo gives us?
created_at: 2025-05-28T12:14:05Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,132 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export const DEFAULT_MAX_LIMIT = 3e4;
+export const DEFAULT_REQUEST_LIMIT = 1e4;
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export async function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const {
+    data = {},
+    query = {},
+    headers = {},
+    parseAs = 'json',
+    maxRedirections,
+  } = opts;
+
+  const requestPath = `/api/${apiVersion}/${path}/`;
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    maxRedirections,
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
+}
+
+export async function requestWithPagination(state, path, options = {}) {
+  const results = [];
+
+  let {
+    pageSize = DEFAULT_REQUEST_LIMIT,
+    start,
+    limit,
+    ...otherOptions
+  } = options;
+
+  const isUsingDefaultLimit = limit === undefined;
+  const maxResults = limit ?? DEFAULT_MAX_LIMIT;
+
+  let isFirstRequest = true;
+  let requestOptions = { query: { start, limit: pageSize }, ...otherOptions };
+  let shouldFetchMoreContent = false;
+  const didUserPassLimit = Boolean(limit);
path: packages/kobotoolbox/src/util.js
user: login: mtuchi
body: Thanks for spotting that out
created_at: 2025-05-29T16:05:27Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
path: packages/kobotoolbox/test/util.test.js
user: login: mtuchi
body: I have clean up the tests mentioned above, please give it another review @josephjclark 
created_at: 2025-05-29T16:09:35Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
+    const { results, next, previous, requestCount } = responseWithPagination(
+      items,
+      {},
+      { url: 'www' }
path: packages/kobotoolbox/test/util.test.js
user: login: mtuchi
body: Each test need a url , the `responseWithPagination` helper is expecting you to pass `url` every time it's called 
created_at: 2025-05-29T16:11:01Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,132 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export const DEFAULT_MAX_LIMIT = 3e4;
path: packages/kobotoolbox/src/util.js
user: login: mtuchi
body: I have rename it to `DEFAULT_LIMIT`
created_at: 2025-05-29T16:12:17Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,132 @@
+import { composeNextState } from '@openfn/language-common';
+import {
+  request as commonRequest,
+  makeBasicAuthHeader,
+  logResponse,
+} from '@openfn/language-common/util';
+
+export const DEFAULT_MAX_LIMIT = 3e4;
+export const DEFAULT_REQUEST_LIMIT = 1e4;
+
+export function prepareNextState(state, response) {
+  const { body, ...responseWithoutBody } = response;
+  return {
+    ...composeNextState(state, body),
+    response: responseWithoutBody,
+  };
+}
+export async function request(state, method, path, opts = {}) {
+  const { baseUrl, apiVersion, username, password } = state.configuration;
+  const {
+    data = {},
+    query = {},
+    headers = {},
+    parseAs = 'json',
+    maxRedirections,
+  } = opts;
+
+  const requestPath = `/api/${apiVersion}/${path}/`;
+  const authHeaders = makeBasicAuthHeader(username, password);
+
+  const options = {
+    body: data,
+    headers: {
+      'Content-Type': 'application/json',
+      ...authHeaders,
+      ...headers,
+    },
+    query: {
+      format: 'json',
+      ...query,
+    },
+    maxRedirections,
+    parseAs,
+    baseUrl,
+  };
+
+  return commonRequest(method, requestPath, options).then(logResponse);
+}
+
+export async function requestWithPagination(state, path, options = {}) {
+  const results = [];
+
+  let {
+    pageSize = DEFAULT_REQUEST_LIMIT,
+    start,
+    limit,
+    ...otherOptions
+  } = options;
+
+  const isUsingDefaultLimit = limit === undefined;
+  const maxResults = limit ?? DEFAULT_MAX_LIMIT;
+
+  let isFirstRequest = true;
+  let requestOptions = { query: { start, limit: pageSize }, ...otherOptions };
+  let shouldFetchMoreContent = false;
+  const didUserPassLimit = Boolean(limit);
+
+  do {
+    requestOptions.query ??= {};
+
+    if (!isNaN(start)) {
+      requestOptions.query.start = start;
+    }
+
+    if (didUserPassLimit || !isFirstRequest) {
+      // If there's an explicit limit or page size,
+      // or this is not the first request,
+      // set the limit in the URL
+      requestOptions.query.limit = Math.min(
+        pageSize || maxResults,
+        maxResults - results.length
+      );
+    } else if (!isNaN(pageSize)) {
+      requestOptions.query.limit = pageSize;
+    }
+
+    const response = await request(state, 'GET', path, requestOptions);
+
+    if (response.body?.results) {
+      results.push(...response.body.results);
+
+      if (!start) {
path: packages/kobotoolbox/src/util.js
user: login: mtuchi
body: Good question 🤔 , Looking at the code now, it's not important. So i have removed that check `if(!start)`
created_at: 2025-05-29T16:13:38Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,429 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_REQUEST_LIMIT, DEFAULT_MAX_LIMIT } from '../src/util';
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+const mockSubmissions = (formId, submissions, opts = {}) => {
+  const { pageSize, ...otherOpts } = opts;
+
+  const path = `/api/v2/assets/${formId}/data/`;
+  const url = `${configuration.baseUrl}${path}`;
+  if (otherOpts.query && typeof otherOpts.query !== 'string') {
+    otherOpts.query = JSON.stringify(otherOpts.query);
+  }
+  testServer
+    .intercept({
+      path,
+      method: 'GET',
+      query: {
+        format: 'json',
+        ...otherOpts,
+      },
+    })
+    .reply(
+      200,
+      (req, res) => {
+        const { query, origin } = req;
+        const results = responseWithPagination(
+          submissions,
+          {
+            limit: otherOpts.limit,
+            start: otherOpts.start,
+          },
+          {
+            url,
+            defaulLimit: pageSize,
+          }
+        );
+        return results;
+      },
+      {
+        ...jsonHeaders,
+      }
+    );
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleResponse = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+  const formId = 'aXecHjmbATuF6iGFmvBLBX';
+
+  it('should return submissions within default page limits', async () => {
+    let requestCount = 0;
+    const totalRequests = Math.ceil(
+      sampleResponse.length / DEFAULT_REQUEST_LIMIT
+    );
+    console.log({ totalRequests });
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleResponse,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(totalRequests);
+    const { data } = await execute(getSubmissions('aXecHjmbATuF6iGFmvBLBX'))(
+      state
+    );
+    expect(data).to.eql(sampleResponse);
+    expect(requestCount).to.eql(totalRequests);
path: packages/kobotoolbox/test/adaptor.test.js
user: login: mtuchi
body: Okay i have clean up the test a bit to remove dynamic assertions 
created_at: 2025-05-29T16:22:35Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
+    const { results, next, previous, requestCount } = responseWithPagination(
+      items,
+      {},
+      { url: 'www' }
path: packages/kobotoolbox/test/util.test.js
user: login: josephjclark
body: It could be defaulted though? we're not really using the value in a meaningful way
created_at: 2025-05-30T06:48:14Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,336 @@
+import { enableMockClient } from '@openfn/language-common/util';
+import { expect } from 'chai';
+import * as util from '../src/util';
+import { responseWithPagination } from './helper';
+
+const { DEFAULT_MAX_LIMIT, DEFAULT_REQUEST_LIMIT } = util;
+
+const testServer = enableMockClient('https://util-test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://util-test.kobotoolbox.org',
+};
+
+const defaultObjects = [
+  { uid: '1' },
+  { uid: '2' },
+  { uid: '3' },
+  { uid: '4' },
+  { uid: '5' },
+  { uid: '6' },
+];
+
+// TODO - test URL building in a set of separate tests
+describe('responseWithPagination', () => {
+  const items = [1, 2, 3];
+
+  it('use default limit and default start', () => {
+    const { results, next, previous, requestCount } = responseWithPagination(
+      items,
+      {},
+      { url: 'www' }
path: packages/kobotoolbox/test/util.test.js
user: login: mtuchi
body: Yes, actually that's what i was thinking. Let me do that 
created_at: 2025-05-30T06:49:41Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,334 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_PAGE_SIZE, DEFAULT_LIMIT } from '../src/util';
+const defaultLimit = DEFAULT_LIMIT;
+const pageLimit = DEFAULT_PAGE_SIZE;
+
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleData = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+
+  it('should not return more items than maximum page size', async () => {
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: What does page size have to do with it?

Isn't this `should not return more items than the  default limit` ?

You then need to duplicate the test with `should return all items if limit is Infinity`

Then I think I have a comment on the infinity test at the end
created_at: 2025-05-30T06:53:58Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,334 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_PAGE_SIZE, DEFAULT_LIMIT } from '../src/util';
+const defaultLimit = DEFAULT_LIMIT;
+const pageLimit = DEFAULT_PAGE_SIZE;
+
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleData = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+
+  it('should not return more items than maximum page size', async () => {
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: Oh but wait, we already have a test called `should be able to explicity get more items than the default max limit`.  So you don't need that "duplicate test".
created_at: 2025-05-30T06:54:51Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,334 @@
+import { expect } from 'chai';
+import { enableMockClient } from '@openfn/language-common/util';
+
+import {
+  execute,
+  getForms,
+  getSubmissions,
+  getDeploymentInfo,
+} from '../src/Adaptor';
+import { responseWithPagination } from './helper';
+import { DEFAULT_PAGE_SIZE, DEFAULT_LIMIT } from '../src/util';
+const defaultLimit = DEFAULT_LIMIT;
+const pageLimit = DEFAULT_PAGE_SIZE;
+
+const testServer = enableMockClient('https://test.kobotoolbox.org');
+const jsonHeaders = {
+  headers: {
+    'Content-Type': 'application/json',
+  },
+};
+const configuration = {
+  username: 'test',
+  password: 'strongpassword',
+  apiVersion: 'v2',
+  baseUrl: 'https://test.kobotoolbox.org',
+};
+
+describe('getSubmissions', () => {
+  const state = { configuration };
+  const sampleData = [
+    { uid: '1', name: 'Ruh' },
+    { uid: '2', name: 'Shay' },
+  ];
+  let originalLog;
+  let consoleOutput = [];
+  beforeEach(() => {
+    originalLog = console.warn;
+    console.warn = (...args) => consoleOutput.push(...args);
+  });
+
+  it('should not return more items than maximum page size', async () => {
+    let requestCount = 0;
+    const items = Array.from({ length: 4e4 }, (_, i) => ({
+      uid: String(i),
+    }));
+
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            items,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(3);
+    const { data } = await execute(getSubmissions('aXecHjmbATuF6iGFmvBLBX'))(
+      state
+    );
+
+    expect(requestCount).to.eql(3);
+    expect(data.length).to.eql(3e4);
+    expect(items.length).to.greaterThan(data.length);
+  });
+
+  it('should only return 1 item if the limit is 1', async () => {
+    const limit = 1;
+    const mockData = sampleData.slice(0, limit);
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          const results = responseWithPagination(
+            sampleData,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+          return results;
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(1);
+
+    const { data } = await execute(
+      getSubmissions('aXecHjmbATuF6iGFmvBLBX', { limit })
+    )(state);
+
+    expect(data).to.eql(mockData);
+    expect(requestCount).to.eql(1);
+    expect(data.length).to.eql(1);
+  });
+
+  it('should be able to explicity get more items than the default max limit', async () => {
+    const manyObjects = Array.from({ length: defaultLimit + 2 }, (_, i) => ({
+      uid: String(i),
+    }));
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            manyObjects,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(4);
+
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX', {
+      limit: 4e4,
+    })(state);
+
+    expect(requestCount).to.eql(4);
+    expect(data.length).to.eql(3e4 + 2);
+    expect(data.length).to.greaterThan(defaultLimit);
+  });
+  it('should allow custom pageSize', async () => {
+    const state = { configuration };
+    const pageSize = 1;
+    const mockData = Array.from({ length: 10 }, (_, i) => ({
+      uid: String(i),
+    }));
+
+    let requestCount = 0;
+    testServer
+      .intercept({
+        path: /\/api\/v2\/assets\/aXecHjmbATuF6iGFmvBLBX\/data/,
+        method: 'GET',
+      })
+      .reply(
+        200,
+        req => {
+          requestCount++;
+          const { query, origin, path } = req;
+          return responseWithPagination(
+            mockData,
+            {
+              limit: query.limit,
+              start: query.start,
+            },
+            {
+              url: `${origin}${path}`,
+              defaultLimit: 1,
+            }
+          );
+        },
+        {
+          ...jsonHeaders,
+        }
+      )
+      .times(10);
+
+    const { data } = await getSubmissions('aXecHjmbATuF6iGFmvBLBX', {
+      pageSize,
+    })(state);
+
+    expect(data.length).to.eql(10);
+    expect(requestCount).to.eql(10);
+  });
+
+  it('should return all items if limit is greater than the number of items (ie, limit = Infinity)', async () => {
path: packages/kobotoolbox/test/adaptor.test.js
user: login: josephjclark
body: I would simply rename this test to "should get all items if limit is Infinity"
created_at: 2025-05-30T06:56:44Z
author_association: COLLABORATOR

PR comments:
user: login: josephjclark
created_at: 2025-04-24T18:20:47Z
body: @mtuchi I'm having a couple of problems here, but I think I need to put this down until tomorrow.

First off, I'm getting a 301 back from my test server, which is triggering an error. Not sure what that's about.

Secondly, I've confused myself over the handling of query. It  doesn't help that kobo's docs on this are awful. I'm not convinced we got this right in the last release. Trying to verify that.

Thirdly, I don't think we  need `limit` AND `max` here  - but that depends a bit on the previous point.

The reason we have both in openMRS is that the properties `options` all get appended to the url - apart from the stuff documented as `openfn only`. So if the user sets `limit`, it'll set the `limit` query parameter, which has meaning to OpenMRS. And I didn't want to conflict that meaning, so I introduced max.

Here, I don't think the `options` get appended to the URL as query parameters. `options.query` expects a query string or object (which gets encoded into the query param).

Which means that `options.limit` is free, and doesn't have to map to a kobo parameter. And so we don't need limit AND max.

I need to re-read this in the morning and verify all this. It gets confusing :sweat_smile: 
author_association: COLLABORATOR
user: login: josephjclark
created_at: 2025-04-25T08:49:41Z
body: From call:
1) Yes, let's drop `max`. All options are openfn options
2)  Add `?format=json` to all requests (I think this will fix the 301)
author_association: COLLABORATOR
user: login: josephjclark
created_at: 2025-05-09T09:26:32Z
body: @mtuchi why is this in blocked? Do you want to pair on it for an hour so that we can get it merged?
author_association: COLLABORATOR
user: login: mtuchi
created_at: 2025-05-09T10:05:32Z
body: > @mtuchi why is this in blocked? Do you want to pair on it for an hour so that we can get it merged?

I think i got `requestWIthPagination` wrong, Will definitely need to pair 
author_association: COLLABORATOR
user: login: mtuchi
created_at: 2025-05-27T16:15:36Z
body: Hiya @josephjclark , I have pushed some changes in this PR. I will take tomorrow on what needs to be updated on changeset and clean up the PR description 
author_association: COLLABORATOR
user: login: mtuchi
created_at: 2025-05-29T16:34:59Z
body: Hiya @josephjclark i have cleanup unit test quite a lot, please have another look 
author_association: COLLABORATOR