Project: YugabyteDB (App Definition and Development / Database)
File: cursors.md

title: Cursors [YSQL] headerTitle: Cursors linkTitle: Cursors description: Explains what a cursor is and how you create and use a cursor with either SQL or PL/pgSQL. [YSQL]. menu: v2.18: identifier: cursors parent: api-ysql weight: 50 type: docs {{< warning title="YSQL currently supports only fetching rows from a cursor consecutively in the forward direction." >}} See the section below. {{< /warning >}} <! to_do Fix up the x-ref's when targets exist in the PL/pgSQL section for the 'bound refcursor' and open > This section explains: what a cursor is; how you can manipulate a cursor explicitly, when your use case calls for this, using either a SQL API or a PL/pgSQL API. The SQL API is exposed by the , , , and statements. Each of these specifies its cursor using an identifier for the cursor's name. The functionally equivalent PL/pgSQL API is exposed by the executable statements open, move, fetch, and close. Each of these specifies its cursor using an identifier for the name of a value of the dedicated data type refcursor. You can declare a refcursor, just as you declare other variables, in the PL/pgSQL source code's . Notice that one flavor of the declaration syntax lets you specify the defining subquery (see below) for the underlying cursor that it denotes. Alternatively, you can specify the data type of a PL/pgSQL subprogram's formal argument as refcursor. The value of a variable or argument whose data type is refcursor is text and is simply the name of the underlying cursor that it denotes. A cursor is an artifact that you create with the SQL statementor with the equivalent PL/pgSQL _open_ statement. A _cursor's_ duration is limited to the lifetime of the session that creates it, it is private to that session, and it is identified by a bare name that must conform to the usual rules for SQL names like those of tables, schemas, and so on. In the sense that it's session-private, that its name isn't schema-qualified, that it has at most session duration, and that it has no explicit owner, it resembles a prepared statement. A session's currently extant _cursors_ are listed in the catalog view. A cursor is defined by a subquery (typically a select statement, but you can use a values statement) and it lets you fetch the rows from the result set that the subquery defines one-at-a-time without (in favorable cases) needing ever to materialize the entire result set in your application's client backend server processi.e. the process that this query lists: ```plpgsql select applicationname, backendtype from pgstatactivity where pid = pgbackendpid(); ``` {{< note title="The internal implementation of a cursor." >}} &nbsp; You don't need to understand this. But it might help you to see how cursors fit into the bigger picture of how the processing of SQL statements like select, values, insert, update, and delete is implemented. (These statements can all be used as the argument of the prepare statementand for this reason, they will be referred to here as preparable statements. At the lowest level, the implementation of all of SQL processing is implemented in PostgreSQL using C. And YSQL uses the same C code. The execution of a preparable statement uses C structure(s) that hold information, in the backend server process, like the SQL statement text, its parsed representation, its execution plan, and so

Further, when the statement is currently being executed, other information is held, like the values of actual arguments that have been bound to placeholders in the SQL text and the position of the current row in the result set (when the statement produces one). You can manipulate these internal structures, from client side code, using the API or, at a level of abstraction above that, the API (a.k.a. the ECPG). Moreover, engineers who implement PostgreSQL itself can use the . These APIs have schemes that let the programmer ask for the entire result set from a subquery in a single round trip. And they also have schemes that let you ask for the result set row-by-row, or in batches of a specified size. See, for example, the libpq subsection . In a more abstract, RDBMS-independent, discussion of the SQL processing of the statements that correspond to PostgreSQL's preparable statements, the term "cursor" is used to denote these internal structures. (You'll notice this, for example, with Oracle Database.) But in PostgreSQL, and therefore in YSQL, a cursor is a direct exposure into SQL and PL/pgSQL (as language features) of just a subset of the internal mechanisms for the processing of preparable statements: the values statement and the select statement when it has no data-modifying side-effects. Here's a counter-example where the select statement does modify data. You can begin a select statement using a that can include a data-modifying statement like insert as long as it has a returning clause. (This ability lets you implement, for example, multi-table insert.) First create a temporary table as the destination for an insert: ```plpgsql create table pg_temp.count(n int not null); ``` Prepare and execute a subquery thus: ```plpgsql prepare stmt as with c(n) as ( insert into pg_temp.count(n) select count(*) from pg_class where relkind = 'v' returning n) select relname from pg_class where relkind = 'v'; execute stmt; ``` This works fineso far. It shows lots of view names and populates the pgtemp.count table with the count. Now try to declare a cursor using the same subquery_: ```plpgsql declare cur cursor for with c(n) as ( insert into pg_temp.count(n) select count(*) from pg_class where relkind = 'v' returning n) select relname from pg_class where relkind = 'v'; ``` It fails with the 0A000 error: DECLARE CURSOR must not contain data-modifying statements in WITH. {{< /note >}} Tautologically, then, a cursor is an artifact that is characterized thus: It is created with the declare SQL statement (or the equivalent open PL/pgSQL statement). Its maximum duration is either the session in which it's created or the transaction in which it's created, according to a choice that you make when you create it. Its lifetime can be terminated deliberately with the close SQL statement or the same-spelled PL/pgSQL statement. It is defined by its name, its subquery and some other boolean attributes. Its name and its other attributes are listed in the pgcursors_ catalog view. It lets you fetch consecutive rows, either one at a time, or in batches whose size you choose, from the result set that its subquery defines; It supports the move SQL statement, and the same-spelled PL/pgSQL statement, to let you specify any row, by its position, within the result set as the current row. It supports the fetch SQL statement, and the same-spelled PL/pgSQL statement, that, in one variant, lets you fetch the row at the current position or a row at any other position relative to the current position (either ahead of it or behind

It supports another variant of the fetch statement (but here only in SQL) that lets you fetch a specified number of rows either forward from and including the row immediately after the current position or backward from and including the row immediately before the current position. The "current position" notion is defined by imagining that the cursor's defining subquery always includes this select list item: ```plpgsql rownumber() over() as currentposition ``` Here, over() spans the entire result set. If the overall query has no order by clause, then over() has no such clause either. But if the overall query does have an order by clause, then over() has the same order by clause within its parentheses. When you execute the move statement, the current position is left at the result to which you moved. And execute the fetch statement (fetching one or several rows in either the forward or the backward direction) the current position is left at the last-fetched result. Set the psql variables db and u to, respectively, a convenient sandbox database and a convenient test role that has connect and create on that database. Then create a trivial helper that will delay the delivery of each row from a subquery's result set: ```plpgsql \c :db :u drop schema if exists s cascade; create schema s; create function s.sleeponesecond() returns boolean set searchpath = pgcatalog, pg_text language plpgsql as $body$ begin perform pg_sleep(1.0); return true; end; $body$; ``` Use it to define a view whose result set is ten rows: ```plpgsql create view s.ten_rows(v) as select a.v from generateseries(1, 10) as a(v) where s.sleepone_second(); ``` Now execute a query in the obvious way at the ysqlsh prompt: ```plpgsql select v from s.ten_rows; ``` It takes about ten seconds before you see any resultsand then you see all ten rows effectively instantaneously. (Use the \\timing on meta-command to time it.) In other words, all ten rows were first materialized in the backend server process's memory before being passed, in a single round trip, to the application. Create another helper function to fetch one row from a cursor and to return its value together with the time taken to fetch it; ```plpgsql create function s.next_row(cur in refcursor) returns text set searchpath = pgcatalog, pg_text language plpgsql as $body$ declare t0 float8 not null := 0; t1 float8 not null := 0; t int not null := 0; v int; begin t0 := extract(epoch from clock_timestamp()); fetch next from cur into v; t1 := extract(epoch from clock_timestamp()); t := (round(t1 - t0)*1000.0)::int; return rpad(coalesce(v::text, '<no row>'), 9)||'-- '||to_char(t, '9999')||' ms'; end; $body$; ``` Now use a cursor to fetch the ten rows one by one: ```plpgsql \t on start transaction; declare "My Cursor" no scroll cursor without hold for select v from s.ten_rows; select s.next_row('My Cursor'); select s.next_row('My Cursor'); select s.next_row('My Cursor'); select s.next_row('My Cursor'); select s.next_row('My Cursor'); select s.next_row('My Cursor'); select s.next_row('My Cursor'); select s.next_row('My Cursor'); select s.next_row('My Cursor'); select s.next_row('My Cursor'); select s.next_row('My Cursor'); rollback; \t off ``` Now you see the rows delivered one by one, every second. This is the result. (Blank lines were removed

```output 1 -- 1000 ms 2 -- 1000 ms 3 -- 1000 ms 4 -- 1000 ms 5 -- 1000 ms 6 -- 1000 ms 7 -- 1000 ms 8 -- 1000 ms 9 -- 1000 ms 10 -- 1000 ms <no row> -- 0 ms ``` When you execute "select v from tenrows" ordinarily using ysqlsh, you have to wait until the entire result set has been materialized in the memory of its backend server process before it's delivered to the client application as a unit. This incurs a memory usage cost as well as a time-delay irritation. But when you declare a cursor for that select_ statement, you materialize the results one row at a time and deliver each to the client as soon as its available. When you use this approach, no more than a single row needs ever to be concurrently materialized in the backend server process's memory. In real applications, you'll use the piecewise result set delivery that a cursor supports only when the result set is vast; and you'll fetch it in batches of a suitable size: small enough that the backend server process's memory isn't over-consumed; but large enough that the round-trip time doesn't dominate the overall cost of fetching a batch. A cursor can be declared either as so-called holdableor not. See the account of the with hold or without hold choice in the section for the statement. Try this: ```plpgsql \c :db :u select count(*) from pg_cursors; start transaction; declare "Not Holdable" cursor without hold for select 17; declare "Is Holdable" cursor with hold for select 42; select name, isholdable::text from pgcursors order by name; commit; select name, isholdable::text from pgcursors order by name; \c :db :u select count(*) from pg_cursors; ``` An invocation of "select count(*) from pgcursors", immediately after starting a session, will inevitably report that no cursors exist. The first pgcursors query (within the ongoing transaction) produces this result: ```output name | is_holdable --+- Is Holdable | true Not Holdable | false ``` And the pgcursors_ query immediately after committing the transaction produces this result: ```output name | is_holdable --+- Is Holdable | true ``` In other words, a non-holdable cursor will vanish when the transaction within which it was declared endseven if the transaction is committed. Because a non-holdable cursor cannot exist outside of an ongoing transaction, this attempt: ```plpgsql declare "Not Holdable" cursor without hold for select 17; ``` causes the 25P01 error: DECLARE CURSOR can only be used in transaction blocks. The wording is slightly confusing because this causes no such error: ```output declare "Is Holdable" cursor with hold for select 42; ``` See the section . The assumption is that you're running ysqlsh with the default setting of 'on' for the psql variable AUTOCOMMIT. Notice that the transactional behavior of a cursor differs critically from that of a prepared statement: ```plpgsql \c :db :u select count(*) from pgpreparedstatements; start transaction; prepare stmt as select 42; rollback; select name from pgpreparedstatements; ``` Like is the case for cursors, an invocation of "select count(*) from pgpreparedstatements", immediately after starting a session, will inevitably report that no prepared statements exist. But even when a statement is prepared within a transaction that is rolled back, it continues to exist after that until either the session ends or it is deallocated. (If you create a holdable cursor, within an on going transaction and then roll back the transaction, then it vanishes.) {{< tip title="Open a holdable cursor in its own transaction and close it as soon as you have finished using it." >}} When, as is the normal practice, you don't subvert the behavior that automatically commits a SQL statement that is not executed within an explicitly started transaction, you'll probably declare, move in and fetch from a holdable cursor "ordinarily"i.e. without explicitly starting, and ending, transactions. A holdable cursor consumes resources because it always caches its defining subquery's entire result

Therefore (and especially in a connection-pooling scheme, you should close a holdable cursor as soon as you have finished using it. {{< /tip >}} {{< note title="A holdable cursor is most useful when you intend to move or to fetch in the backward direction but YSQL does not yet support this." >}} See the section below. {{< /note >}} When you choose, at cursor creation time, either the scroll or the no scroll options, the result of your choice is shown in the isscrollable column in the pgcursors view. Try this: ```plpgsql start transaction; declare "Not Scrollable" no scroll cursor without hold for select g.v from generate_series(1, 5) as g(v); declare "Is Scrollable" scroll cursor without hold for select g.v from generate_series(1, 5) as g(v); select name, isscrollable::text from pgcursors order by name; rollback; ``` This is the result: ```output name | is_scrollable -+ Is Scrollable | true Not Scrollable | false ``` The term of art scrollable reflects a rather unusual meaning of scrollability. In, for example, discussions about GUIs, scrolling means moving forwards or backwards within a window or, say, a list. However: When pgcursors.isscrollable is false, this means that you can change the current position in the cursor's result set (using either move or as a consequence of fetch) only in the forward direction. When pgcursors.isscrollable is true, this means that you can change the current position in the cursor's result set both in the forward direction and in the backward direction. In other words: When you create a cursor and specify no scroll, you're saying that you will allow changing the current position in the result set in only the forward direction. When you create a cursor and specify scroll, you're saying that you will allow changing the current position in the result set in both the forward direction and the backward direction. Notice that your choice with the move statement, to change the current position by just a single row or by many rows is an orthogonal choice to the direction in which you move. Similarly, your choice with the fetch statement, to fetch just a single row or many rows is an orthogonal choice to the direction in which you fetch. There is no way to create a cursor so that changing the current position in the result set by more than one row, except by consecutive fetches, is prevented. <a name="specify-no-scroll-or-scroll-explicitly"></a> {{< tip title="Always specify either 'no scroll' or 'scroll' explicitly." >}} If you specify neither no scroll nor scroll when you create a cursor, then you don't get an error. However, the outcome is that sometimes backwards movement in the result set is allowed, and sometimes it causes the 55000 error: cursor can only scan forward. Yugabyte recommends that you always specify your scrollability choice explicitly to honor the requirements that you must meet. Notice that while remains open, your only viable choice is no scroll. {{< /tip >}} Do this: ```plpgsql set clientminmessages = error; start transaction; declare cur no scroll cursor without hold for select g.v from generate_series(1, 5) as g(v); fetch next from cur; move relative 2 in cur; fetch forward 2 from cur; ``` So far, this runs without error and produces these results, as expected: ```output 1 4 5 ``` Now do this: ```plpgsql fetch relative 0 from cur; rollback; ``` The fetch causes the 55000 error: cursor can only scan forward because fetching a cursor's current row tautologically does not move the current position forward and is therefore not allowed when you create the cursor using no

Now do this: ```plpgsql set clientminmessages = error; start transaction; declare cur scroll cursor without hold for select g.v from generate_series(1, 5) as g(v); -- These are the same four statements that the "no scroll" demo used. -- But now, "fetch relative 0" succeeds. fetch next from cur; move relative 2 in cur; fetch forward 2 from cur; fetch relative 0 from cur; -- Add these statements. fetch backward from cur; fetch backward all from cur; move last in cur; fetch relative 0 from cur; move first in cur; fetch relative 0 from cur; rollback; ``` Now every fetch and move statement succeeds. These are the results: ```output 1 4 5 5 4 3 2 1 5 1 ``` The result set for a with hold cursor is always cached when the transaction that creates it commits. The result set for a without hold cursor might, or might not, be cached. If the execution plan for the cursor's defining subquery can be executed in either the forward or the backward direction, then the result set will not be cached. But if the plan can be executed only in the forward direction, then the result set must be cached if you specify scroll when you create the cursor. PostgreSQL, and therefore YSQL, do not expose metadata to report whether or not a cursor's result set is cached. Nor does the documentation for either RDBMS attempt to specify the rules that determine whether caching will be done. However, it's possible to reason, about certain specific select statements, that their plans cannot be executed backward. For example the plan for a query that includes rownumber() in the select list cannot be run backward because the semantics of rownumber() is to assign an incrementing rank to each new row in the result set as it is produced when the plan is executed in the forward directionand the planner cannot predict how many rows will be produced to allow rownumber() to be calculated by decrementing from this for each successive row when the plan is run backward. (If the select statement has no order by, then the rows are produced in physical order_ (i.e. in an order that's determined by how the table data is stored). <a name="physical-order-cannot-be-predicted"></a> {{< note title="The physical order cannot be predicted." >}} The physical order cannot be predicted; and it might even change between repeat executions of the same select statement. However, if you use a cluster on a developer laptop and ensure that the backend process that supports the session that you use to do the tests is the only process whose type is client backend, then it's very likely indeed that successive repeats of the same select statement will produce the same physical orderat least over the timescale of typical ad hoc experiments. {{< /note >}} You can, however, support pedagogy by including a user-defined function in the where clause that always returns true and that uses raise info to report when it's invoked. First, do this set-up. All the caching tests will use it: ```plpgsql set clientminmessages = error; drop schema if exists s cascade; create schema s; create table s.t(k serial primary key, v int not null); insert into s.t(v) select generate_series(1, 5); create view s.v(pos, v) as select row_number() over(), v from s.t; create function

in int) returns boolean set searchpath = pgcatalog, pg_temp volatile language plpgsql as $body$ begin raise info 'f() invoked'; return i = i; end; $body$; ``` Do this: ```plpgsql start transaction; declare cur no scroll cursor with hold for select pos, v from s.v where s.f(v); ``` It completes silently without error. Now do this: ```plpgsql commit; ``` This is when you see the raise info outputfive times in total, i.e. once for each row that's tested: ```output INFO: f() invoked INFO: f() invoked INFO: f() invoked INFO: f() invoked INFO: f() invoked ``` This demonstrates that the result set has been cached. Now fetch all the rows and close the cursor; ```plpgsql fetch all from cur; close cur; ``` The fetch all statement brings this SQL output: ```output 1 | 5 2 | 1 3 | 4 4 | 2 5 | 3 ``` Do this: ``` start transaction; declare cur no scroll cursor without hold for select pos, v from s.v where s.f(v); fetch next from cur; fetch next from cur; fetch next from cur; fetch next from cur; fetch next from cur; rollback; ``` It's easier to distinguish the raise info output and the SQL output from the statements that bring these if you save this code to, say, t.sql, and then execute it at the ysqlsh prompt. This is what you see. ```output INFO: f() invoked 1 | 5 INFO: f() invoked 2 | 1 INFO: f() invoked 3 | 4 INFO: f() invoked 4 | 2 INFO: f() invoked 5 | 3 ``` Notice that the same v values are paired with the same pos values as with the "with hold", "no scroll" cursor test. But here, nothing suggests that results are cachedand they don't need to be because the cursor doesn't allow moving backwards in the result set. (However, you can't design a test to demonstrate that results are not cached.) Do this: ``` start transaction; declare cur scroll cursor without hold for select pos, v from s.v where s.f(v); -- Moving forward. fetch next from cur; fetch next from cur; fetch next from cur; fetch next from cur; fetch next from cur; -- Moving backward. fetch prior from cur; move absolute 3 in cur; fetch relative 0 from cur; move absolute 1 in cur; fetch relative 0 from cur; rollback; ``` Apart from replacing no scroll with scroll in the declaration of the cursor, the code is identical, through all the "Moving forward fetches, to that for the "without hold", "no scroll" cursor test. And the output is identical, too. Now continue with the tests that move backwards: You no longer see any raise info output because the entire result set is now cached. Here are the SQL results: ```output 4 | 2 3 | 4 1 | 5 ``` Notice that you see the same v values for the same pos values as before. This is required by the semantics of moving forward and backward through the unique result set that the cursor's subquery defines. The output from the previous test shows that the result set is cached incrementally. This variant demonstrates it more vividly by interleaving forward and backward fetches and moves. First, increase the number of rows in s.t and inspect s.v: ```plpgsql delete from s.t; insert into s.t(v) select generate_series(1, 10); select pos, v from s.v; ``` This is the result: ```output 1 | 6 2 | 8 3 | 7 4 | 10 5 | 1 6 | 2 7 | 4 8 | 5 9 | 3 10 | 9 ``` You might see the v values in a different order. See the note

Now do this: ```plpgsql start transaction; declare cur scroll cursor without hold for select pos, v from s.v where s.f(v); ``` This succeeds silently without error. Now do this: ```plpgsql fetch forward 3 from cur; fetch backward 2 from cur; ``` It causes this output: ```output INFO: f() invoked INFO: f() invoked INFO: f() invoked 1 | 6 2 | 8 3 | 7 2 | 8 1 | 6 ``` We don't see any raise info output when we fetch the rows with pos = 2 and pos = 1 again because they're already cached. Now do this: ```plpgsql fetch forward 5 from cur; fetch backward 3 from cur; ``` It causes this output: ``` INFO: f() invoked INFO: f() invoked INFO: f() invoked 2 | 8 3 | 7 4 | 10 5 | 1 6 | 2 5 | 1 4 | 10 3 | 7 ``` Because we're now seeing the three rows with pos = 4, pos = 5, and pos = 6 for the first time, we see the raise info output three times. Then, when we revisit the rows with pos = 5, pos = 4, and pos = 3, we don't see any raise info output because, again, they're already cached. Now do this: ```plpgsq fetch forward 5 from cur; fetch backward 1 from cur; ``` It causes this output: ```plpgsql INFO: f() invoked INFO: f() invoked 4 | 10 5 | 1 6 | 2 7 | 4 8 | 5 7 | 4 ``` Here, we're seeing the rows with pos = 7 and pos = 8 for the first timewhich brings the raise info output twice. And then, because we've already (just) seen the row with pos 7, there's now no raise info output. Finally, do this: ```plpgsql move absolute 3 in cur; fetch forward 7 from cur; rollback; ``` It causes this output: ```outpit INFO: f() invoked INFO: f() invoked 4 | 10 5 | 1 6 | 2 7 | 4 8 | 5 9 | 3 10 | 9 ``` At this stage, we've seen all rows in the result set but those with pos = 9 and pos = 10, so we see the raise info for just each of those two rows. {{< warning title="YSQL currently supports only fetching rows from a cursor consecutively in the forward direction." >}} &nbsp; tracks the problem that the SQL statements fetch and move, together with their PL/pgSQL counterparts, don't work reliably. This is reflected by warnings that are drawn under these circumstances: Every move flavor causes the 0A000 warning with messages like "MOVE not supported yet". Many fetch flavors draw the 0A000 warning with messages like "FETCH FIRST not supported yet", "FETCH LAST not supported yet", "FETCH BACKWARD not supported yet", and the like. These are the only fetch flavors that do not draw a warning: fetch next bare fetch fetch :N bare fetch forward fetch forward :N fetch all and fetch forward all :N must be a positive integer. Notice that in many tests, and warning messages notwithstanding, the code has exactly the same effect in YSQL as it does in vanilla PostgreSQL. And if you set the clientminmessages run-time parameter to error, then the apparently spurious warnings are suppressed. This is the case for all the tests shown on this page. However, you must not trust this for production code while remains open. You should not suppress warnings, and you should not use any operation that draws the _0A000_ warning. {{< /warning >}}