Project: YugabyteDB (App Definition and Development / Database)
File: common-code.md

title: Common code for traversing all kinds of graph headerTitle: Common code for traversing all kinds of graph linkTitle: Common code description: This section presents common helper code for traversing all kinds of graph menu: v2.18: identifier: common-code parent: traversing-general-graphs weight: 20 type: docs Make sure that you have run before you run the following code. Once you've done this, you can leave all of the common code that you create by following the steps described below in place while you run the various code examples in the following sections that show how to traverse the various different kinds of graph. The main value of this function is to bring readability. Without it, all of the SQL implementations described in the following sections would be cluttered with, in some cases, three occurrences of this expression: ``` a.path[cardinality(a.path)] ``` It's a little shorter, and much easier to discern the meaning of, this expression: ``` terminal(a.path) ``` Moreover, in some cases, an index will be needed on a table of "path" values but this statement: ```plpgsql create unique index terminal_unq on <the path table>(path[cardinality(path)]) ``` fails with this generic error: ``` 42601: syntax error ``` But this works just fine: ``` create unique index terminal_unq on paths(terminal(path)); ``` So it's essential to use the function "terminal()" here. The same thinking applies for this: ``` create unique index filteredpathsshortestpathunq on shortest_paths(start(path), terminal(path)); ``` ```plpgsql drop function if exists start cascade; drop function if exists terminal cascade; create function start(path in text[]) returns text immutable language plpgsql as $body$ begin return path[1]; end; $body$; create function terminal(path in text[]) returns text immutable language plpgsql as $body$ begin return path[cardinality(path)]; end; $body$; ``` It will be useful to insert the paths that the code examples for the various kinds of graph generate into a table so that subsequent ad hoc queries can be run on these results. For example, it is interesting to generate the set of shortest paths to the distinct reachable nodes (see ). Because all these tables have the same shape and constraints, it is best to use dynamic SQL issued from a procedure to create them. Notice that, in order to ensure that every computed path is unique, a unique index is created on the "path" column. However, currently prevents the direct creation of an index on an array. The workaround is to create the index on the `text` typecast of the array. However, the nave attempt: ``` create index i1 on t1((arr::text)); ``` fails for a different reason: ``` ERROR: 42P17: functions in index expression must be marked IMMUTABLE ``` (The "typecast" functionality is given by a special kind of built-in function that is not considered to be immutable.) So a jacket function, "pathastext()", that can be marked as immutable is used. This is risk free because the `text` typecast of an array is reliably deterministic and free of

```plpgsql drop function if exists pathastext(text[]) cascade; create function pathastext(path in text[]) returns text immutable language plpgsql as $body$ begin return path::text; end; $body$; drop procedure if exists createpathtable(text, boolean) cascade; create procedure createpathtable(name in text, temp in boolean) language plpgsql as $body$ declare drop_table constant text := ' drop table if exists ? cascade'; create_table constant text := ' create table ?( k serial primary key, path text[] not null)'; createtemptable constant text := ' create temporary table ?( k serial primary key, path text[] not null)'; cache_sequence constant text := ' alter sequence ?kseq cache 100000'; begin execute replace(drop_table, '?', name); case temp when true then execute replace(createtemptable, '?', name); else execute replace(create_table, '?', name); end case; execute replace(cache_sequence, '?', name); end; $body$; ``` You'll use the procedure to create the following tables: "rawpaths"_. This is the target table for the paths that each of the code examples for the various kinds of graph generates. "shortestpaths". This is the target table for the paths produced by procedure _. "unqcontainingpaths". This is the target table for the paths produced by procedure . "temppaths", and "previouspaths". These tables are used by the approach that uses direct SQL, issued from a PL/pgsql stored procedure, that implements what the recursive CTE does rather than an actual `WITH` clause. See the section . First create the "rawpaths" table and optionally add a column and create a trigger on the table so that the outcome of each successive repeat of the code that implements the recursive term can be traced to help the developer see how the code works. (It has this effect only for the implementation of the "findpaths()" procedure that implements early-path-pruning.) Because the purpose of the trigger is entirely pedagogical, because it is not needed by the substantive "findpaths()" logic, and because its presence brings a theoretical performance drag, you should either use the script that adds the extra tracing column and trigger for the "rawpaths" table or use the script that simply creates the "rawpaths"_ table bareaccording to your purpose. EITHER: ```plpgsql call createpathtable('raw_paths', false); alter table rawpaths add column repeatnr int; drop function if exists rawpathstrg_f() cascade; create function rawpathstrg_f() returns trigger language plpgsql as $body$ declare max_iteration constant int := ( select coalesce(max(repeatnr), null, -1) + 1 from rawpaths); begin update rawpaths set repeatnr = maxiteration where repeatnr is null; return new; end; $body$; create trigger rawpathstrg after insert on raw_paths for each statement execute function rawpathstrg_f(); ``` OR: ```plpgsql call createpathtable('raw_paths', false); drop function if exists rawpathstrg_f() cascade; ``` You can see that you can run either one of `cr-raw-paths-with-tracing.sql` or `cr-raw-paths-no-tracing.sql` at any time to get the regime that you need. Normally, you'll set up the "rawpaths"_ table for tracing. But when your purpose is to compare the times for different approaches, you'll set it up without tracing code. Now create the other tables: ```plpgsql call createpathtable('shortest_paths', false); call createpathtable('unqcontainingpaths', false); call createpathtable('temp_paths', true); call createpathtable('previous_paths', true); create unique index shortestpathsstartterminalunq on shortest_paths(start(path), terminal(path)); ``` The solution to the problem needs only the shortest path to all those actors who have a transitive "both acted in the same movie" relationship to Kevin Bacon. But, in general, there will be many paths that reflect this transitive relationship. Indeed, there might even be two or more paths that each has the same shortest length. The "restricttoshortestpaths()"_ procedure finds just one shortest path to each reachable node from the set of all paths to reachable

When there do exist two or more shortest paths to the same node, it selects the first one in the path sorting order. The advantage of this scheme over picking one of the contenders randomly is that the result is deterministic. This allows for a meaningful comparison between the result from running two overall analyses in two different databases. This is crucial when the aim is to confirm that PostgreSQL and YugabyteDB produce the same result from the same starting data, where, without a reliable ordering scheme, differences in physical data storage might produce different actual orders of results. Notice how ordinary (non-recursive) CTEs are used, just as functions and procedures are used in procedural programming, to encapsulate and name distinct steps in the code. Try to implement the logic without using this technique. The exercise will very vividly highlight the expressive value that the `WITH` clause provides. ```plpgsql drop procedure if exists restricttoshortest_paths(text, text, boolean) cascade; -- Restrict the paths from the start node to every other node -- to just the shortest path to each distinct node. create procedure restricttoshortest_paths( intab in text, outtab in text, append in boolean default false) language plpgsql as $body$ declare stmt constant text := ' with -- For readability. Define cardinality and end_node as view columns. a1(k, cardinality, end_node, path) as ( select k, cardinality(path), path[cardinality(path)], path from ?in_tab), -- In general, there is more than one path to any end_node. Define the -- minumum cardinality for each end_node as a pair of view columns. a2(cardinality, end_node) as ( select min(cardinality), end_node from a1 group by end_node), -- There might still be more than one path to a particular end_node -- where each of these has the minimum cardinality. Pick up the path -- for each of these following the GROUP BY to allow just one of these -- to be picked arbitrarily. a3(path, cardinality, end_node) as ( select a1.path, cardinality, end_node from a1 inner join a2 using(cardinality, end_node)), -- Pick just one path among the possibly several minimum cardinality paths -- to each end_node. a4(path, cardinality, end_node) as ( select min(path), cardinality, end_node from a3 group by cardinality, end_node) -- Finally, pick up the actual path to each arbitrily selected end_node to which the -- path has the minimum cardinality. No need to include the end_node column now because -- it is anyway shown as the final node in the path. insert into ?out_tab(path) select path from a4 inner join a1 using(path, cardinality, end_node)'; begin case append when false then execute 'delete from '||out_tab; else null; end case; execute replace(replace(stmt, '?intab', intab), '?outtab', outtab); end; $body$; ``` This procedure is derived trivially from the procedure by just a few obvious substitutions: "longest" for "shortest", "maximum" for "minimum" , and "max()" for "min()": ```plpgsql drop procedure if exists restricttolongest_paths(text, text, boolean) cascade; -- Restrict the paths from the start node to every other node -- to just the longest path to each distinct node. create procedure restricttolongest_paths( intab in text, outtab in text, append in boolean default false) language plpgsql as $body$ declare stmt constant text := ' with -- For readability. Define cardinality and end_node as view

a1(k, cardinality, end_node, path) as ( select k, cardinality(path), path[cardinality(path)], path from ?in_tab), -- In general, there is more than one path to any end_node. Define the -- maximum cardinality for each end_node as a pair of view columns. a2(cardinality, end_node) as ( select max(cardinality), end_node from a1 group by end_node), -- There might still be more than one path to a particular end_node -- where each of these has the maximum cardinality. Pick up the path -- for each of these following the GROUP BY to allow just one of these -- to be picked arbitrarily. a3(path, cardinality, end_node) as ( select a1.path, cardinality, end_node from a1 inner join a2 using(cardinality, end_node)), -- Pick just one path among the possibly several maximum cardinality paths -- to each end_node. a4(path, cardinality, end_node) as ( select max(path), cardinality, end_node from a3 group by cardinality, end_node) -- Finally, pick up the actual path to each arbitrily selected end_node to which the -- path has the maximum cardinality. No need to include the end_node column now because -- it is anyway shown as the final node in the path. insert into ?out_tab(path) select path from a4 inner join a1 using(path, cardinality, end_node)'; begin case append when false then execute 'delete from '||out_tab; else null; end case; execute replace(replace(stmt, '?intab', intab), '?outtab', outtab); end; $body$; ``` Suppose that the paths "n1 > n2", "n1 > n2 > n3", "n1 > n2 > n3 > n4", and "n1 > n2 > n3 > n4 > n5" were all found by a call like this: ``` call findpaths(startnode => 'n1'); ``` See, for example, . By construction, all of the paths that it finds start at the node "n1". You can see that the path "n1 > n2 > n3 > n4 > n5" contains each of the shorter paths "n1 > n2", "n1 > n2 > n3", and "n1 > n2 > n3 > n4". This means that all of the useful information about the paths that have been found, in this example, is conveyed by just the longest containing path. It can sometimes be useful, therefore, to summarize the many results produced by a call to `findpaths()` by listing only the set of unique longest containing paths. The procedure `restricttounqcontaining_paths()` finds this set. It relies on the . ```plpgsql drop procedure if exists restricttounqcontainingpaths(text, text, boolean) cascade; -- Filter the input set of paths to set of longest paths -- that jointly contain all the other paths. create procedure restricttounqcontainingpaths( intab in text, outtab in text, append in boolean default false) language plpgsql as $body$ declare stmt constant text := ' with -- Cartesian product restricted to give all possible -- longer path with shorter path combinations. eachpathwithallshorter_paths as ( select a1.path as longerpath, a2.path as shorterpath from ?intab as a1, ?intab as a2 where cardinality(a1.path) > cardinality(a2.path)), -- Identify each shorter path that is contained by -- its longer path partner. contained_paths as ( select shorterpath as containedpath from eachpathwithallshorter_paths where longerpath @> shorterpath) -- Filter out the contained

insert into ?out_tab(path) select path from ?in_tab where path not in ( select containedpath from containedpaths )'; begin case append when false then execute 'delete from '||out_tab; else null; end case; execute replace(replace(stmt, '?intab', intab), '?outtab', outtab); end; $body$; ``` The table function "listpaths()"_ achieves what you could achieve with an ordinary top-level SQL statement if you edited it before each execution to use the name of the target table of interest. This is a canonical use case for dynamic SQL. prevents you using dynamic SQL for a `SELECT` statement that returns more than one row. The function enables a simple workaround. Notice the use of "translate" and "replace" on the text literal representation of a `text script. ```plpgsql drop function if exists list_paths(text) cascade; create function list_paths(tab in text) returns table(t text) language plpgsql as $body$ declare -- Recall that when you address an array element that falls outside of its bounds, -- you get a `NULL` result. And recall that `NULLS FIRST` is the default sorting order. stmt constant text := $$ with a(r, c, p) as ( select row_number() over w, cardinality(path), path from ? window w as (order by path[1], cardinality(path), path[2], path[3], path[4], path[5], path[6])) select array_agg( lpad(r::text, 6) ||' '|| lpad(c::text, 11) ||' '|| replace(translate(p::text, '{}', ''), ',', ' > ') order by p[1], cardinality(p), p[2], p[3], p[4], p[5], p[6]) from a $$; results text[] not null := '{}'; begin t := 'path # cardinality path'; return next; t := ' -- -'; return next; execute replace(stmt, '?', tab) into results; foreach t in array results loop return next; end loop; end; $body$; ``` In some cases, the result from one of the implementations of the procedure that finds all the paths from a specified start node finds only the shortest paths. In these cases, the output of the procedure is identical to its input. The following procedure tests that this is the case. ```plpgsql drop procedure if exists assertshortestpathssameasrawpaths() cascade; create procedure assertshortestpathssameasrawpaths() language plpgsql as $body$ declare n int not null := 999999; begin with restrictededexceptraw as ( select path from shortest_paths except select path from raw_paths), rawexceptrestricted as ( select path from raw_paths except select path from shortest_paths), filteredexceptrawunionrawexceptfiltered as ( select path from restrictededexceptraw union all select path from rawexceptrestricted) select count(*) into n from filteredexceptrawunionrawexceptfiltered; assert n = 0, 'unexpected'; end; $body$; ``` The code that the section presents needs to compare the times that alternative implementations take to complete. The \\timing on meta-command doesn't help here because it reports the time after every individual statement and, on Unix-like operating systems, does this using stderr. The \\o meta-command doesn't redirect stderr to the spool file. So you need a SQL scheme to start a stopwatch and later to read it. The section within the overall major section shows you how to implement a SQL stopwatch that allows you to start it with the procedure startstopwatch() before starting what you want to time and to read it with the function stopwatchreading() when what you want to time finishes. The result of selecting this function goes to the spool file along with all other query results. The dedicated section links to a .zip of all of the code so that you can install it for any purpose. However, so that the downloadable code for the overall section, , is self-contained, the necessary stopwatch code is simply reproduced verbatim within that.