Repository: LLNL/Caliper (language: python)

Issue:
Question: Is there a way to have Caliper output a custom string with the built in configs?
I would like, for example, the runtime-report to print out a string containing a git version string for our library.


Discussion / hints:
Hi @balos1 ,

Currently that's not possible, but it is an interesting idea. We might be able to add that.

However, we have a C++ API that lets you write reports into a custom C++ stream, which might be a workaround. It requires MPI support. Here's an example:

https://software.llnl.gov/Caliper/ConfigManagerAPI.html#_CPPv4N4cali30make_collective_output_channelEPKc

Hope this helps!
-David
Thanks for the reply @daboehme. 

The reasoning for wanting this: we would like to know exactly what version of our library was used when the report was generated. There are certainly ways we could work around this (I will take a look at the link), but it would be a very convenient feature if one could just set an environment variable like `CALI_PKG_REVISION=$(git rev-parse HEAD)` (or through a runtime API option) and have that string in the report.
Ok, I'll try to come up with something.
So just ran across this and, having done something like this myself, figured I'd shared what I think in the context of Caliper the solution could be: a `metadata` service (maybe enabled by default) which registers a bunch of (obviously) metadata at start-up and then there would a `cali_add_metadata` function that users can use to add their own metadata.

E.g. at start-up:

```cpp
// uses strftime
auto _launch_date = get_local_datetime("%D", settings::get_launch_time());
auto _launch_time = get_local_datetime("%H:%M", settings::get_launch_time());
auto _cpu_info    = cpu::get_info();
auto _user        = get_env<std::string>("USER", "nobody");

for(auto&& itr : { "SHELL", "HOME", "PWD" })
{
    auto _var = get_env<std::string>(itr, "");
    if(!_var.empty())
        add_metadata(itr, _var);
}

add_metadata("USER", _user);
add_metadata("LAUNCH_DATE", _launch_date);
add_metadata("LAUNCH_TIME", _launch_time);

// these would be computed in CMake and put into a generated version.h header
add_metadata("CALIPER_VERSION", CALIPER_VERSION_STRING);
add_metadata("CALIPER_GIT_DESCRIBE", CALIPER_GIT_DESCRIBE);
add_metadata("CALIPER_GIT_REVISION", CALIPER_GIT_REVISION);

add_metadata("CPU_MODEL", _cpu_info.model);
add_metadata("CPU_VENDOR", _cpu_info.vendor);
add_metadata("CPU_FEATURES", _cpu_info.features);
add_metadata("CPU_FREQUENCY", _cpu_info.frequency);
add_metadata("HW_CONCURRENCY", threading::hw_concurrency());
add_metadata("HW_PHYSICAL_CPU", threading::hw_physicalcpu());

for(int i = 0; i < 3; ++i)
{
    auto              _cache_size = cpu::cache_size::get(i + 1);
    std::stringstream _cache_lvl;
    _cache_lvl << "HW_L" << (i + 1) << "_CACHE_SIZE";
    add_metadata(_cache_lvl.str(), _cache_size);
}
```
We actually have a metadata collection API much like the one suggested by @jrmadsen . Metadata attributes are called "global attributes" in Caliper, and you can use the `cali_set_global_[int|double|string]_byname` calls to set them, e.g. `cali_set_global_string_byname("MY_VERSION", "v4.2");`. However, the preferred way to do this is through the [Adiak](https://github.com/LLNL/Adiak) library and the `adiak_import` service. Adiak has functions to grab some common info like launch date, as well as key-value style functions to add custom application-specific metadata.

There is also a way to import metadata from environment variables: the `env` service. I totally forgot about that 😄. You can use `CALI_ENV_EXTRA` to specify the names of environment variables that should be imported, e.g.

```
$ export MY_GIT_REVISION="xyzabc"
$ export MY_VERSION="v4.2"
$ CALI_SERVICES_ENABLE=env,... CALI_ENV_EXTRA="MY_GIT_REVISION,MY_VERSION" ./app
``` 

Caliper adds this metadata to the machine-readable .json and .cali outputs, but we're still missing a way to print them together with the human-readable tree or table reports. This shouldn't be difficult though, and I'll add that. 

Fix patch:
diff --git a/include/caliper/reader/QuerySpec.h b/include/caliper/reader/QuerySpec.h
index 19833df52..4a43cb3d2 100644
--- a/include/caliper/reader/QuerySpec.h
+++ b/include/caliper/reader/QuerySpec.h
@@ -105,7 +105,7 @@ struct QuerySpec
             Default, User
         }                        opt;       ///< Default or user-defined formatter
         FunctionSignature        formatter; ///< The formatter to use. Signatures provided by FormatProcessor.
-        std::vector<std::string> args;      ///< Arguments to the formatter.
+        std::map<std::string, std::string> kwargs; ///< Arguments to the formatter.
     };
 
     struct PreprocessSpec {
diff --git a/src/caliper/controllers/CMakeLists.txt b/src/caliper/controllers/CMakeLists.txt
index d1d5a2a22..d84980fae 100644
--- a/src/caliper/controllers/CMakeLists.txt
+++ b/src/caliper/controllers/CMakeLists.txt
@@ -6,7 +6,8 @@ set(CALIPER_CONTROLLERS_SOURCES
   controllers/HatchetSampleProfileController.cpp
   controllers/OpenMPReportController.cpp
   controllers/RuntimeReportController.cpp
-  controllers/controllers.cpp)
+  controllers/controllers.cpp
+  controllers/util.cpp)
 
 list(APPEND CALIPER_RUNTIME_SOURCES ${CALIPER_CONTROLLERS_SOURCES})
 set(CALIPER_RUNTIME_SOURCES ${CALIPER_RUNTIME_SOURCES} PARENT_SCOPE)
diff --git a/src/caliper/controllers/CallpathSampleReportController.cpp b/src/caliper/controllers/CallpathSampleReportController.cpp
index 20417e306..fcd452b83 100644
--- a/src/caliper/controllers/CallpathSampleReportController.cpp
+++ b/src/caliper/controllers/CallpathSampleReportController.cpp
@@ -3,6 +3,8 @@
 
 #include "caliper/caliper-config.h"
 
+#include "util.h"
+
 #include "caliper/ChannelController.h"
 #include "caliper/ConfigManager.h"
 
@@ -33,10 +35,6 @@ class CallpathSampleReportController : public cali::ChannelController
                 + std::to_string(1.0/freq)
                 + ") as \"Time (sec)\" unit sec";
 
-            std::string format = std::string("tree(source.function#callpath.address,")
-                + opts.get("max_column_width", "48").to_string()
-                + ")";
-
             // Config for second aggregation step in MPI mode (cross-process aggregation)
             std::string cross_select =
                   " min(scount) as \"Min time/rank\" unit sec"
@@ -52,6 +50,8 @@ class CallpathSampleReportController : public cali::ChannelController
             if (have_pthread)
                 config()["CALI_SERVICES_ENABLE"].append(",pthread");
 
+            std::string format = util::build_tree_format_spec(config(), opts, "path-attributes=source.function#callpath.address");
+
             if (use_mpi) {
                 config()["CALI_SERVICES_ENABLE"   ].append(",mpi,mpireport");
                 config()["CALI_MPIREPORT_FILENAME"] = opts.get("output", "stderr").to_string();
@@ -116,7 +116,7 @@ const char* callpath_sample_report_spec =
     "{"
     " \"name\"        : \"callpath-sample-report\","
     " \"description\" : \"Print a call-path sampling profile for the program\","
-    " \"categories\"  : [ \"metric\", \"output\" ],"
+    " \"categories\"  : [ \"metric\", \"output\", \"treeformatter\" ],"
     " \"services\"    : [ \"callpath\", \"sampler\", \"symbollookup\", \"trace\" ],"
     " \"config\"      : "
     "   { \"CALI_CHANNEL_FLUSH_ON_EXIT\"      : \"false\""
@@ -132,11 +132,6 @@ const char* callpath_sample_report_spec =
     "   \"name\": \"aggregate_across_ranks\","
     "   \"type\": \"bool\","
     "   \"description\": \"Aggregate results across MPI ranks\""
-    "  },"
-    "  {"
-    "   \"name\": \"max_column_width\","
-    "   \"type\": \"int\","
-    "   \"description\": \"Maximum column width in the tree display\""
     "  }"
     " ]"
     "}";
diff --git a/src/caliper/controllers/CudaActivityReportController.cpp b/src/caliper/controllers/CudaActivityReportController.cpp
index 4e76e958e..321e719fc 100644
--- a/src/caliper/controllers/CudaActivityReportController.cpp
+++ b/src/caliper/controllers/CudaActivityReportController.cpp
@@ -3,6 +3,8 @@
 
 #include "caliper/caliper-config.h"
 
+#include "util.h"
+
 #include "caliper/ChannelController.h"
 #include "caliper/ConfigManager.h"
 
@@ -52,6 +54,8 @@ class CudaActivityReportController : public cali::ChannelController
                     = std::string("cupti.kernel.name as Kernel,") + cross_select;
             }
 
+            std::string format = util::build_tree_format_spec(config(), opts);
+
             if (use_mpi) {
                 config()["CALI_SERVICES_ENABLE"   ].append(",mpi,mpireport");
                 config()["CALI_MPIREPORT_FILENAME"] = opts.get("output", "stderr").to_string();
@@ -65,7 +69,7 @@ class CudaActivityReportController : public cali::ChannelController
                     opts.build_query("cross", {
                             { "select",   cross_select  },
                             { "group by", groupby },
-                            { "format",   "tree"  }
+                            { "format",   format  }
                         });
             } else {
                 config()["CALI_SERVICES_ENABLE"   ].append(",report");
@@ -74,7 +78,7 @@ class CudaActivityReportController : public cali::ChannelController
                     opts.build_query("local", {
                             { "select",   serial_select },
                             { "group by", groupby },
-                            { "format",   "tree"  }
+                            { "format",   format  }
                         });
             }
 
@@ -116,7 +120,7 @@ const char* controller_spec =
     " \"name\"        : \"cuda-activity-report\","
     " \"description\" : \"Record and print CUDA activities (kernel executions, memcopies, etc.)\","
     " \"categories\"  : [ \"output\", \"region\", \"cuptitrace.metric\" ],"
-    " \"services\"    : [ \"aggregate\", \"cupti\", \"cuptitrace\", \"event\" ],"
+    " \"services\"    : [ \"aggregate\", \"cupti\", \"cuptitrace\", \"event\", \"treeformatter\" ],"
     " \"config\"      : "
     "   { \"CALI_CHANNEL_FLUSH_ON_EXIT\"        : \"false\","
     "     \"CALI_EVENT_ENABLE_SNAPSHOT_INFO\"   : \"false\","
diff --git a/src/caliper/controllers/RuntimeReportController.cpp b/src/caliper/controllers/RuntimeReportController.cpp
index 2a5260a4b..4c1b3b9e4 100644
--- a/src/caliper/controllers/RuntimeReportController.cpp
+++ b/src/caliper/controllers/RuntimeReportController.cpp
@@ -3,6 +3,8 @@
 
 #include "caliper/caliper-config.h"
 
+#include "util.h"
+
 #include "caliper/ChannelController.h"
 #include "caliper/ConfigManager.h"
 
@@ -43,13 +45,7 @@ class RuntimeReportController : public cali::ChannelController
                 pmetric = "inclusive_percent_total(sum#sum#time.duration)";
             }
 
-            std::string format = "tree";
-
-            if (opts.is_set("max_column_width")) {
-                format = "(prop:nested,";
-                format.append(opts.get("max_column_width").to_string());
-                format.append(")");
-            }
+            std::string format = util::build_tree_format_spec(config(), opts);
 
             // Config for second aggregation step in MPI mode (cross-process aggregation)
             std::string cross_select =
@@ -122,7 +118,7 @@ const char* runtime_report_spec =
     "{"
     " \"name\"        : \"runtime-report\","
     " \"description\" : \"Print a time profile for annotated regions\","
-    " \"categories\"  : [ \"metric\", \"output\", \"region\" ],"
+    " \"categories\"  : [ \"metric\", \"output\", \"region\", \"treeformatter\" ],"
     " \"services\"    : [ \"aggregate\", \"event\", \"timestamp\" ],"
     " \"config\"      : "
     "   { \"CALI_CHANNEL_FLUSH_ON_EXIT\"      : \"false\","
@@ -142,11 +138,6 @@ const char* runtime_report_spec =
     "   \"name\": \"aggregate_across_ranks\","
     "   \"type\": \"bool\","
     "   \"description\": \"Aggregate results across MPI ranks\""
-    "  },"
-    "  {"
-    "   \"name\": \"max_column_width\","
-    "   \"type\": \"int\","
-    "   \"description\": \"Maximum column width in the tree display\""
     "  }"
     " ]"
     "}";
diff --git a/src/caliper/controllers/controllers.cpp b/src/caliper/controllers/controllers.cpp
index bcc9c02e0..df92d6f52 100644
--- a/src/caliper/controllers/controllers.cpp
+++ b/src/caliper/controllers/controllers.cpp
@@ -662,6 +662,18 @@ const char* builtin_option_specs =
     " \"description\" : \"Adiak import categories. Comma-separated list of integers.\","
     " \"type\"        : \"string\","
     " \"category\"    : \"adiak\""
+    "},"
+    "{"
+    " \"name\"        : \"max_column_width\","
+    " \"type\"        : \"int\","
+    " \"description\" : \"Maximum column width in the tree display\","
+    " \"category\"    : \"treeformatter\""
+    "},"
+    "{"
+    " \"name\"        : \"print.metadata\","
+    " \"type\"        : \"bool\","
+    " \"description\" : \"Print program metadata (Caliper globals and Adiak data)\","
+    " \"category\"    : \"treeformatter\""
     "}"
     "]";
 
diff --git a/src/caliper/controllers/util.cpp b/src/caliper/controllers/util.cpp
new file mode 100644
index 000000000..7ce0611a1
--- /dev/null
+++ b/src/caliper/controllers/util.cpp
@@ -0,0 +1,30 @@
+// Copyright (c) 2021, Lawrence Livermore National Security, LLC.
+// See top-level LICENSE file for details.
+
+#include "util.h"
+
+#include "../../services/Services.h"
+
+#include <algorithm>
+
+using namespace cali;
+
+std::string util::build_tree_format_spec(config_map_t& config, const ConfigManager::Options& opts, const char* initial)
+{
+    std::string format = initial;
+
+    if (opts.is_set("max_column_width"))
+        format.append("column-width=").append(opts.get("max_column_width").to_string());
+    if (opts.is_enabled("print.metadata")) {
+        auto avail_services = services::get_available_services();
+        bool have_adiak =
+            std::find(avail_services.begin(), avail_services.end(), "adiak_import") != avail_services.end();
+
+        if (have_adiak)
+            config["CALI_SERVICES_ENABLE"].append(",adiak_import");
+
+        format.append(format.length() > 0 ? "," : "").append("print-globals");
+    }
+
+    return std::string("tree(") + format + ")";
+}
diff --git a/src/caliper/controllers/util.h b/src/caliper/controllers/util.h
new file mode 100644
index 000000000..6da687269
--- /dev/null
+++ b/src/caliper/controllers/util.h
@@ -0,0 +1,24 @@
+// Copyright (c) 2021, Lawrence Livermore National Security, LLC.
+// See top-level LICENSE file for details.
+
+#pragma once
+#ifndef CALI_CONTROLLERS_UTIL_H
+#define CALI_CONTROLLERS_UTIL_H
+
+#include "caliper/ConfigManager.h"
+
+#include <string>
+
+namespace cali
+{
+
+namespace util
+{
+
+std::string build_tree_format_spec(config_map_t&, const cali::ConfigManager::Options&, const char* initial = "");
+
+}
+
+}
+
+#endif // CALI_CONTROLLERS_UTIL_H
\ No newline at end of file
diff --git a/src/reader/CMakeLists.txt b/src/reader/CMakeLists.txt
index 816690477..065ca4cc9 100644
--- a/src/reader/CMakeLists.txt
+++ b/src/reader/CMakeLists.txt
@@ -14,6 +14,7 @@ set(CALIPER_READER_SOURCES
   QueryProcessor.cpp
   QuerySpec.cpp
   RecordSelector.cpp
+  SnapshotTableFormatter.cpp
   SnapshotTree.cpp
   TableFormatter.cpp
   TreeFormatter.cpp
diff --git a/src/reader/CalQLParser.cpp b/src/reader/CalQLParser.cpp
index 1b88c60d1..da06ee4b1 100644
--- a/src/reader/CalQLParser.cpp
+++ b/src/reader/CalQLParser.cpp
@@ -35,6 +35,29 @@ get_definition_id(const std::string& w, const QuerySpec::FunctionSignature* defs
     return defs[retid].name ? retid : -1;
 }
 
+std::pair<std::string, std::string> get_keyval(const std::string& str)
+{
+    std::istringstream is(str);
+
+    std::string first = util::read_word(is, "=");
+    char c = util::read_char(is);
+
+    if (c == '=') {
+        std::string val = util::read_word(is);
+        return std::make_pair(std::move(first), std::move(val));
+    } else {
+        return std::make_pair(std::string(), std::move(first));
+    }
+}
+
+bool is_in_strlist(const std::string& s, const char** list)
+{
+    for (const char** p = list; *p; ++p)
+        if (s == *p)
+            return true;
+    return false;
+}
+
 } // namespace [anonymous]
 
 
@@ -107,7 +130,7 @@ struct CalQLParser::CalQLParserImpl
         }
 
         do {
-            std::string str = util::read_word(is, ",;=<>()\n");
+            std::string str = util::read_word(is, ",;()\n");
             c = util::read_char(is);
 
             if (!str.empty() && (c == ',' || c == ')'))
@@ -123,6 +146,33 @@ struct CalQLParser::CalQLParserImpl
         return ret;
     }
 
+    std::map<std::string, std::string>
+    make_kwargs(const QuerySpec::FunctionSignature& sig, const std::vector<std::string>& args, std::istream& is)
+    {
+        std::map<std::string, std::string> ret;
+        int n = std::min(static_cast<int>(args.size()), sig.max_args);
+
+        for (int i = 0; i < n; ++i) {
+            auto p = get_keyval(args[i]);
+            if (p.first.empty()) {
+                if (is_in_strlist(p.second, sig.args))
+                    ret[p.second] = std::string("true"); // support boolean flags like in JsonFormatter
+                else
+                    ret[sig.args[i]] = p.second; // use name of positional argument
+            } else {
+                if (is_in_strlist(p.first, sig.args)) {
+                    ret[p.first] = p.second;
+                } else {
+                    set_error("Unknown argument " + p.first + " for " + sig.name, is);
+                    ret.clear();
+                    break;
+                }
+            }
+        }
+
+        return ret;
+    }
+
     void
     parse_aggregate(std::istream& is) {
         const QuerySpec::FunctionSignature* defs = Aggregator::aggregation_defs();
@@ -180,7 +230,7 @@ struct CalQLParser::CalQLParserImpl
         else {
             spec.format.opt       = QuerySpec::FormatSpec::User;
             spec.format.formatter = defs[i];
-            spec.format.args      = args;
+            spec.format.kwargs    = make_kwargs(defs[i], args, is);
         }
     }
 
diff --git a/src/reader/FormatProcessor.cpp b/src/reader/FormatProcessor.cpp
index 1cff50cf1..f60467d9c 100644
--- a/src/reader/FormatProcessor.cpp
+++ b/src/reader/FormatProcessor.cpp
@@ -20,9 +20,9 @@ namespace
 {
 
 const char* format_kernel_args[] = { "format", "title" };
-const char* tree_kernel_args[]   = { "path-attributes", "column-width" };
-const char* table_kernel_args[]  = { "column-width" };
-const char* json_kernel_args[]   = { "layout", "pretty", "quote-all", "separate-nested" };
+const char* tree_kernel_args[]   = { "path-attributes", "column-width", "print-globals" };
+const char* table_kernel_args[]  = { "column-width", "print-globals" };
+const char* json_kernel_args[]   = { "object", "pretty", "quote-all", "separate-nested", "records", "split" };
 
 enum FormatterID {
     Cali        = 0,
@@ -37,7 +37,7 @@ enum FormatterID {
 const QuerySpec::FunctionSignature formatters[] = {
     { FormatterID::Cali,      "cali",       0, 0, nullptr },
     { FormatterID::Cali,      "csv",        0, 0, nullptr }, // keep old "csv" name for backwards compatibility
-    { FormatterID::Json,      "json",       0, 4, json_kernel_args },
+    { FormatterID::Json,      "json",       0, 6, json_kernel_args },
     { FormatterID::Expand,    "expand",     0, 0, nullptr },
     { FormatterID::Format,    "format",     1, 2, format_kernel_args },
     { FormatterID::Table,     "table",      0, 1, table_kernel_args  },
diff --git a/src/reader/JsonFormatter.cpp b/src/reader/JsonFormatter.cpp
index 498d884dc..a08065fc9 100644
--- a/src/reader/JsonFormatter.cpp
+++ b/src/reader/JsonFormatter.cpp
@@ -75,18 +75,18 @@ struct JsonFormatter::JsonFormatterImpl
 
 
     void configure(const QuerySpec& spec) {
-        for (auto arg : spec.format.args) {
-            if (arg == "pretty")
+        for (auto p : spec.format.kwargs) {
+            if (p.first == "pretty")
                 m_opt_pretty = true;
-            else if (arg == "quote-all")
+            else if (p.first == "quote-all")
                 m_opt_quote_all = true;
-            else if (arg == "separate-nested")
+            else if (p.first == "separate-nested")
                 m_opt_sep_nested = true;
-            else if (arg == "records")
+            else if (p.first == "records")
                 m_layout = Records;
-            else if (arg == "split")
+            else if (p.first == "split")
                 m_layout = Split;
-            else if (arg == "object")
+            else if (p.first == "object")
                 m_layout = Object;
         }
 
diff --git a/src/reader/SnapshotTableFormatter.cpp b/src/reader/SnapshotTableFormatter.cpp
new file mode 100644
index 000000000..4baa2affe
--- /dev/null
+++ b/src/reader/SnapshotTableFormatter.cpp
@@ -0,0 +1,100 @@
+#include "SnapshotTableFormatter.h"
+
+#include "caliper/common/CaliperMetadataAccessInterface.h"
+#include "caliper/common/Node.h"
+
+#include "../common/util/format_util.h"
+
+#include <iostream>
+
+using namespace cali;
+
+namespace
+{
+
+struct EntryInfo  {
+    std::string key;
+    std::string val;
+    bool align_right;
+};
+
+struct RecordInfo {
+    std::vector<EntryInfo> entries;
+    size_t max_key_len;
+    size_t max_right_len;
+
+    void add(const CaliperMetadataAccessInterface& db, cali_id_t attr_id, const Variant& data) {
+        Attribute attr = db.get_attribute(attr_id);
+
+        bool align_right =
+            attr.type() == CALI_TYPE_DOUBLE ||
+            attr.type() == CALI_TYPE_INT    ||
+            attr.type() == CALI_TYPE_UINT;
+
+        EntryInfo info { attr.name(), data.to_string(), align_right };
+
+        max_key_len = std::max(max_key_len, info.key.length());
+
+        if (align_right)
+            max_right_len = std::max(max_right_len, info.val.length());
+
+        entries.push_back(info);
+    }
+
+    RecordInfo()
+        : max_key_len { 0 }, max_right_len { 0 }
+        { }
+};
+
+RecordInfo unpack_record(CaliperMetadataAccessInterface& db, const std::vector<Entry>& rec)
+{
+    RecordInfo info;
+
+    for (const Entry& e : rec) {
+        if (e.is_reference()) {
+            for (const Node* node = e.node(); node && node->attribute() != CALI_INV_ID; node = node->parent())
+                info.add(db, node->attribute(), node->data());
+        } else if (e.is_immediate()) {
+            info.add(db, e.attribute(), e.value());
+        }
+    }
+
+    return info;
+}
+
+std::ostream& write_record_data(const RecordInfo& info, std::ostream& os)
+{
+    const size_t MAX_KEY_COL_WIDTH = 24;
+    const size_t MAX_VAL_COL_WIDTH = 52;
+
+    size_t key_width = std::min(info.max_key_len, MAX_KEY_COL_WIDTH);
+    size_t val_width = MAX_VAL_COL_WIDTH;
+
+    int count = 0;
+
+    for (const EntryInfo e : info.entries) {
+        if (count++ > 0)
+            os << "\n";
+
+        util::pad_right(os, util::clamp_string(e.key, key_width), key_width) << ": ";
+
+        if (e.align_right)
+            util::pad_left(os, e.val, info.max_right_len);
+        else
+            os << util::clamp_string(e.val, val_width);
+    }
+
+    return os << "\n";
+}
+
+} // namespace anonymous
+
+namespace cali
+{
+
+std::ostream& format_record_as_table(CaliperMetadataAccessInterface& db, const std::vector<Entry>& rec, std::ostream& os)
+{
+    return ::write_record_data(::unpack_record(db, rec), os);
+}
+
+}
diff --git a/src/reader/SnapshotTableFormatter.h b/src/reader/SnapshotTableFormatter.h
new file mode 100644
index 000000000..f4fd200c1
--- /dev/null
+++ b/src/reader/SnapshotTableFormatter.h
@@ -0,0 +1,24 @@
+// Copyright (c) 2021, Lawrence Livermore National Security, LLC.
+// See top-level LICENSE file for details.
+
+/// \file SnapshotTableFormatter.h
+/// \brief Defines SnapshotTableFormatter class
+
+#ifndef CALI_SNAPSHOTTABLEFORMATTER_H
+#define CALI_SNAPSHOTTABLEFORMATTER_H
+
+#include "caliper/common/Entry.h"
+
+#include <vector>
+
+namespace cali
+{
+
+class CaliperMetadataAccessInterface;
+
+std::ostream&
+format_record_as_table(CaliperMetadataAccessInterface& db, const std::vector<Entry>&, std::ostream& os);
+
+} // namespace cali
+
+#endif
\ No newline at end of file
diff --git a/src/reader/TableFormatter.cpp b/src/reader/TableFormatter.cpp
index 6bd445b64..e352f4dc8 100644
--- a/src/reader/TableFormatter.cpp
+++ b/src/reader/TableFormatter.cpp
@@ -7,6 +7,8 @@
 
 #include "caliper/reader/QuerySpec.h"
 
+#include "SnapshotTableFormatter.h"
+
 #include "caliper/common/Attribute.h"
 #include "caliper/common/CaliperMetadataAccessInterface.h"
 #include "caliper/common/Node.h"
@@ -53,6 +55,8 @@ struct TableFormatter::TableImpl
 
     int                                     m_max_column_width;
 
+    bool                                    m_print_globals;
+
     int column_width(int base) const {
         return std::max(m_max_column_width > 0 ? std::min(base, m_max_column_width) : base, 4);
     }
@@ -95,13 +99,22 @@ struct TableFormatter::TableImpl
 
         // Set max column width
 
-        if (spec.format.args.size() > 0) {
-            bool ok = false;
+        {
+            auto it = spec.format.kwargs.find("column-width");
+            if (it != spec.format.kwargs.end()) {
+                bool ok = false;
+
+                m_max_column_width = StringConverter(it->second).to_int(&ok);
 
-            m_max_column_width = StringConverter(spec.format.args[0]).to_int(&ok);
+                if (!ok)
+                    m_max_column_width = -1;
+            }
+        }
 
-            if (!ok)
-                m_max_column_width = -1;
+        {
+            auto it = spec.format.kwargs.find("print-globals");
+            if (it != spec.format.kwargs.end())
+                m_print_globals = true;
         }
 
         // Fill sort columns
@@ -318,7 +331,7 @@ struct TableFormatter::TableImpl
     }
 
     TableImpl()
-        : m_max_column_width(60)
+        : m_max_column_width(60), m_print_globals(false)
         { }
 };
 
@@ -346,7 +359,10 @@ TableFormatter::process_record(CaliperMetadataAccessInterface& db, const EntryLi
 }
 
 void
-TableFormatter::flush(CaliperMetadataAccessInterface&, std::ostream& os)
+TableFormatter::flush(CaliperMetadataAccessInterface& db, std::ostream& os)
 {
+    if (mP->m_print_globals)
+        format_record_as_table(db, db.get_globals(), os);
+
     mP->flush(os);
 }
diff --git a/src/reader/TreeFormatter.cpp b/src/reader/TreeFormatter.cpp
index f2eca607f..41a13ba15 100644
--- a/src/reader/TreeFormatter.cpp
+++ b/src/reader/TreeFormatter.cpp
@@ -5,6 +5,8 @@
 
 #include "caliper/reader/TreeFormatter.h"
 
+#include "SnapshotTableFormatter.h"
+
 #include "caliper/reader/QuerySpec.h"
 #include "caliper/reader/SnapshotTree.h"
 
@@ -47,6 +49,7 @@ struct TreeFormatter::TreeFormatterImpl
     std::map<std::string, std::string> m_aliases;
 
     bool                     m_use_nested;
+    bool                     m_print_globals;
 
     std::vector<std::string> m_path_key_names;
     std::vector<Attribute>   m_path_keys;
@@ -60,25 +63,37 @@ struct TreeFormatter::TreeFormatterImpl
 
     void configure(const QuerySpec& spec) {
         // set path keys (first argument in spec.format.args)
-        if (spec.format.args.size() > 0)
-            util::split(spec.format.args.front(), ',',
-                        std::back_inserter(m_path_key_names));
+
+        {
+            auto it = spec.format.kwargs.find("path-attributes");
+            if (it != spec.format.kwargs.end())
+                util::split(it->second, ',', std::back_inserter(m_path_key_names));
+        }
 
         // set max column width
-        if (spec.format.args.size() > 1) {
-            bool ok = false;
+        {
+            auto it = spec.format.kwargs.find("column-width");
+            if (it != spec.format.kwargs.end()) {
+                bool ok = false;
 
-            m_max_column_width = StringConverter(spec.format.args[1]).to_int(&ok);
+                m_max_column_width = StringConverter(it->second).to_int(&ok);
 
-            if (!ok) {
-                Log(0).stream() << "TreeFormatter: invalid column width argument \""
-                                << spec.format.args[1] << "\""
-                                << std::endl;
+                if (!ok) {
+                    Log(0).stream() << "TreeFormatter: invalid column width argument \""
+                                    << it->second << "\""
+                                    << std::endl;
 
-                m_max_column_width = -1;
+                    m_max_column_width = -1;
+                }
             }
         }
 
+        {
+            auto it = spec.format.kwargs.find("print-globals");
+            if (it != spec.format.kwargs.end())
+                m_print_globals = true;
+        }
+
         {
             auto it = std::find(m_path_key_names.begin(), m_path_key_names.end(),
                                 "prop:nested");
@@ -337,7 +352,8 @@ struct TreeFormatter::TreeFormatterImpl
     TreeFormatterImpl()
         : m_path_column_width(0),
           m_max_column_width(48),
-          m_use_nested(true)
+          m_use_nested(true),
+          m_print_globals(false)
     { }
 };
 
@@ -362,5 +378,8 @@ TreeFormatter::process_record(CaliperMetadataAccessInterface& db, const EntryLis
 void
 TreeFormatter::flush(CaliperMetadataAccessInterface& db, std::ostream& os)
 {
+    if (mP->m_print_globals)
+        format_record_as_table(db, db.get_globals(), os);
+
     mP->flush(db, os);
 }
diff --git a/src/reader/UserFormatter.cpp b/src/reader/UserFormatter.cpp
index efa86cba8..7a7b2ed79 100644
--- a/src/reader/UserFormatter.cpp
+++ b/src/reader/UserFormatter.cpp
@@ -29,7 +29,7 @@ struct UserFormatter::FormatImpl
 {
     struct Field {
         string    prefix;
-        
+
         string    attr_name;
         Attribute attr;
 
@@ -44,7 +44,7 @@ struct UserFormatter::FormatImpl
 
     OutputStream  m_os;
     std::mutex    m_os_lock;
-    
+
     FormatImpl(OutputStream& os)
         : m_os(os)
         { }
@@ -98,32 +98,33 @@ struct UserFormatter::FormatImpl
 
                 split_string.erase(split_string.begin());
             }
-            
+
             m_fields.push_back(field);
         }
     }
 
     void configure(const QuerySpec& spec) {
-        if (spec.format.args.size() > 0)
-            parse(spec.format.args[0]);
+        auto it = spec.format.kwargs.find("format");
+        if (it != spec.format.kwargs.end())
+            parse(it->second);
     }
-    
+
     void print(CaliperMetadataAccessInterface& db, const EntryList& list) {
         std::ostringstream os;
-        
+
         for (Field f : m_fields) {
             Attribute attr { Attribute::invalid };
-            
+
             {
                 std::lock_guard<std::mutex>
                     g(m_fields_lock);
-                
+
                 if (f.attr == Attribute::invalid)
                     f.attr = db.get_attribute(f.attr_name);
 
                 attr = f.attr;
             }
-            
+
             string str;
 
             if (attr != Attribute::invalid)
@@ -138,7 +139,7 @@ struct UserFormatter::FormatImpl
                     if (!str.empty())
                         break;
                 }
-            
+
             const char whitespace[80+1] =
                 "                                        "
                 "                                        ";
@@ -154,7 +155,7 @@ struct UserFormatter::FormatImpl
                 g(m_os_lock);
 
             std::ostream* real_os = m_os.stream();
-            
+
             *real_os << os.str() <<std::endl;
         }
     }
@@ -166,9 +167,12 @@ UserFormatter::UserFormatter(OutputStream& os, const QuerySpec& spec)
 {
     mP->configure(spec);
 
-    if (spec.format.args.size() > 1) {
-        std::ostream* real_os = os.stream();
-        *real_os << spec.format.args[1] << std::endl;
+    {
+        auto it = spec.format.kwargs.find("title");
+        if (it != spec.format.kwargs.end()) {
+            std::ostream* real_os = os.stream();
+            *real_os << it->second << std::endl;
+        }
     }
 }
 
@@ -177,7 +181,7 @@ UserFormatter::~UserFormatter()
     mP.reset();
 }
 
-void 
+void
 UserFormatter::process_record(CaliperMetadataAccessInterface& db, const EntryList& list)
 {
     mP->print(db, list);
diff --git a/src/services/report/Report.cpp b/src/services/report/Report.cpp
index fa98d7e8b..9f4bfa456 100644
--- a/src/services/report/Report.cpp
+++ b/src/services/report/Report.cpp
@@ -63,12 +63,13 @@ class Report {
 
         db.add_attribute_aliases(spec.aliases);
         db.add_attribute_units(spec.units);
-        db.import_globals(*c, c->get_globals(channel));
 
         c->flush(channel, flush_info, [&queryP,&db](CaliperMetadataAccessInterface& in_db, const std::vector<Entry>& rec){
                 queryP.process_record(db, db.merge_snapshot(in_db, rec));
             } );
 
+        db.import_globals(*c, c->get_globals(channel));
+
         queryP.flush(db);
     }
 
diff --git a/src/tools/cali-query/query_common.cpp b/src/tools/cali-query/query_common.cpp
index 75976b7e9..52e47a5c6 100644
--- a/src/tools/cali-query/query_common.cpp
+++ b/src/tools/cali-query/query_common.cpp
@@ -207,13 +207,11 @@ QueryArgsParser::parse_args(const Args& args)
 
             // Find formatter args (if any)
             for (int i = 0; i < fmtsig->max_args; ++i)
-                if (args.is_set(fmtsig->args[i])) {
-                    m_spec.format.args.resize(i+1);
-                    m_spec.format.args[i] = args.get(fmtsig->args[i]);
-                }
+                if (args.is_set(fmtsig->args[i]))
+                    m_spec.format.kwargs[fmtsig->args[i]] = args.get(fmtsig->args[i]);
 
             // NOTE: This check isn't complete yet.
-            if (m_spec.format.args.size() < static_cast<std::size_t>(fmtsig->min_args)) {
+            if (m_spec.format.kwargs.size() < static_cast<std::size_t>(fmtsig->min_args)) {
                 m_error     = true;
                 m_error_msg = std::string("Insufficient arguments for formatter ") + fmtsig->name;
 


Test patch:
diff --git a/src/reader/test/CMakeLists.txt b/src/reader/test/CMakeLists.txt
index 257a6873b..8c9efe1c1 100644
--- a/src/reader/test/CMakeLists.txt
+++ b/src/reader/test/CMakeLists.txt
@@ -8,7 +8,8 @@ set(CALIPER_READER_TEST_SOURCES
   test_nestedexclusiveregionprofile.cpp
   test_nestedinclusiveregionprofile.cpp
   test_nodebuffer.cpp
-  test_preprocessor.cpp)
+  test_preprocessor.cpp
+  test_snapshottableformatter.cpp)
 
 add_executable(test_caliper-reader
   $<TARGET_OBJECTS:caliper-common>
diff --git a/src/reader/test/test_calqlparser.cpp b/src/reader/test/test_calqlparser.cpp
index 565bb0407..045c8389b 100644
--- a/src/reader/test/test_calqlparser.cpp
+++ b/src/reader/test/test_calqlparser.cpp
@@ -217,30 +217,57 @@ TEST(CalQLParserTest, OrderByClause3) {
 }
 
 TEST(CalQLParserTest, FormatSpec) {
-    CalQLParser p1("FORMAT tree(\"a,bb,ccc\")");
+    {
+        CalQLParser p1("FORMAT tree(\"a,bb,ccc\")");
 
-    EXPECT_FALSE(p1.error()) << "Unexpected parse error: " << p1.error_msg();
+        EXPECT_FALSE(p1.error()) << "Unexpected parse error: " << p1.error_msg();
 
-    QuerySpec q1 = p1.spec();
+        QuerySpec q1 = p1.spec();
 
-    EXPECT_EQ(q1.format.opt, QuerySpec::FormatSpec::User);
-    EXPECT_STREQ(q1.format.formatter.name, "tree");
-    ASSERT_EQ(q1.format.args.size(), 1);
-    EXPECT_EQ(q1.format.args[0], "a,bb,ccc");
+        EXPECT_EQ(q1.format.opt, QuerySpec::FormatSpec::User);
+        EXPECT_STREQ(q1.format.formatter.name, "tree");
+        ASSERT_EQ(q1.format.kwargs.size(), 1);
+        EXPECT_EQ(q1.format.kwargs["path-attributes"], "a,bb,ccc");
+    }
 
-    CalQLParser p2("FORMAT table");
+    {
+        CalQLParser p2("FORMAT table");
 
-    EXPECT_FALSE(p2.error()) << "Unexpected parse error: " << p2.error_msg();
+        EXPECT_FALSE(p2.error()) << "Unexpected parse error: " << p2.error_msg();
 
-    QuerySpec q2 = p2.spec();
+        QuerySpec q2 = p2.spec();
 
-    EXPECT_EQ(q2.format.opt, QuerySpec::FormatSpec::User);
-    EXPECT_STREQ(q2.format.formatter.name, "table");
-    EXPECT_EQ(q2.format.args.size(), 0);
+        EXPECT_EQ(q2.format.opt, QuerySpec::FormatSpec::User);
+        EXPECT_STREQ(q2.format.formatter.name, "table");
+        EXPECT_EQ(q2.format.kwargs.size(), 0);
+    }
 
-    CalQLParser p3("FORMAT tree(\"a,bb,ccc\", ddd, e)");
+    {
+        CalQLParser p3("FORMAT tree(\"a,bb,ccc\", ddd, e)");
+        EXPECT_TRUE(p3.error());
+    }
 
-    EXPECT_TRUE(p3.error());
+    {
+        CalQLParser p4("FORMAT tree(column-width=42)");
+        EXPECT_FALSE(p4.error()) << "Unexpected parse error: " << p4.error_msg();
+
+        QuerySpec q4 = p4.spec();
+        EXPECT_EQ(q4.format.kwargs["column-width"], std::string("42"));
+    }
+
+    {
+        CalQLParser p5("FORMAT tree(glorb=glurb)");
+        EXPECT_TRUE(p5.error());
+    }
+
+    {
+        CalQLParser p6("FORMAT json(object)");
+        QuerySpec q6 = p6.spec();
+        EXPECT_FALSE(p6.error()) << "Unexpected parse error: " << p6.error_msg();
+        EXPECT_STREQ(q6.format.formatter.name, "json");
+        EXPECT_EQ(q6.format.kwargs.size(), 1);
+        EXPECT_FALSE(q6.format.kwargs.find("object") == q6.format.kwargs.end());
+    }
 }
 
 TEST(CalQLParserTest, AggregateClause) {
diff --git a/src/reader/test/test_snapshottableformatter.cpp b/src/reader/test/test_snapshottableformatter.cpp
new file mode 100644
index 000000000..34e086895
--- /dev/null
+++ b/src/reader/test/test_snapshottableformatter.cpp
@@ -0,0 +1,37 @@
+#include "../SnapshotTableFormatter.h"
+
+#include "caliper/reader/CaliperMetadataDB.h"
+
+#include "caliper/common/Node.h"
+
+#include <gtest/gtest.h>
+
+#include <sstream>
+
+using namespace cali;
+
+TEST(SnapshotTableFormatter, Format) {
+    CaliperMetadataDB db;
+
+    Attribute a = db.create_attribute("aaaa", CALI_TYPE_INT,    CALI_ATTR_ASVALUE);
+    Attribute b = db.create_attribute("bb",   CALI_TYPE_UINT,   CALI_ATTR_ASVALUE);
+    Attribute s = db.create_attribute("str",  CALI_TYPE_STRING, CALI_ATTR_DEFAULT);
+
+    IdMap idmap;
+    const Node* node = db.merge_node(101, s.id(), CALI_INV_ID, Variant("a string value"), idmap);
+
+    std::vector<Entry> rec;
+    rec.push_back(Entry(a,   Variant(42)));
+    rec.push_back(Entry(b, Variant(4242)));
+    rec.push_back(Entry(node));
+
+    std::ostringstream os;
+    format_record_as_table(db, rec, os);
+
+    std::string expect =
+        "aaaa :   42 \n"
+        "bb   : 4242 \n"
+        "str  : a string value\n";
+
+    EXPECT_EQ(os.str(), expect);
+}
\ No newline at end of file