Package: pysteed
Steed Open Source ProjectTable of Contents1 Description2 Pre-requisites2.1 g++2.2 CMake2.3 Doxygen2.4 Graphviz2.5 LZ42.6 CLI112.7 Google Test2.8 All in One3 Source Code3.1 Download3.2 Directory Structure4 Build4.1 Build with CMake4.2 Build with Make4.3 Unittest4.4 Install5 Example5.1 Build with Steed shared library5.2 JSON Example Data5.3 Steed Configuration5.4 Run Example5.4.1 Config and init Static5.4.2 Create Database5.4.3 Create Table5.4.4 Parse Data5.4.5 Assemble Data5.4.6 Drop Table and Database1 DescriptionSteed is a columnar storage engine for JSON data. It is designed to be a lightweight, high-performance, and easy-to-use database engine. Steed is written in C++ and we are going to open source the storage engine, including the JSON parser, column assembler, and columnar data storage.2 Pre-requisitesSteed requires the following software to build:C++ compiler with C++11 supportCMakeDoxygenLZ4CLI11Google TestWe have tested Steed on the following platforms:Ubuntu 22.04 or laterCMake 3.11 or laterDoxygen 1.9.1 or laterLZ4 1.9.3 or laterCLI11 2.3.2 or laterGoogle Test 1.8.0 or later2.1 g++Steed requires a C++ compiler with C++11 support. We have tested the following compilers:g++ 7.5.0 or laterTo install g++, use the following command:$sudoapt-getinstallg++2.2 CMakeSteed uses CMake to build. We have tested the following CMake versions:CMake 3.11 or laterTo install CMake, use the following command:$sudoapt-getinstallcmakeOr you can download the latest version of CMake fromCMake.2.3 DoxygenSteed uses Doxygen to generate documentation. We have tested the following Doxygen versions:1.9.1 or laterTo install Doxygen, use the following command:$sudoapt-getinstalldoxygenOr you can download the latest version of Doxygen fromDoxygen.2.4 GraphvizDoxygen uses Graphviz to generate graphs. To install Graphviz, use the following command:$sudoapt-getinstallgraphviz2.5 LZ4Steed uses LZ4 to compress and decompress data. We have tested the following LZ4 versions:1.9.3 or laterTo install LZ4, use the following command:$sudoapt-getinstallliblz4-1liblz4-dev2.6 CLI11Steed uses CLI11 to parse the configuration file. We have tested the following CLI11 versions:2.3.2 or laterWe do contain the CLI11 source code in thesrc/thirdparty/cli11directory and the License of CLI11 is in thesrc/thirdparty/cli11/LICENSEfile. CMake will build CLI11 automatically.2.7 Google TestSteed uses Google Test to run unittests. We have tested the following Google Test versions:1.8.0 or laterWe also contain the Google Test source code in thesrc/thirdparty/googletestdirectory and the License of Google Test is in thesrc/thirdparty/googletest/LICENSEfile. CMake will build Google Test automatically.2.8 All in OneYou can use the following command to install all the pre-requisites:$sudoapt-getinstallgitg++cmakedoxygengraphvizliblz4-1liblz4-devpython3-dev3 Source Code3.1 DownloadSteed is available on Gitlab. You can clone the Steed repository using the following command:$gitclonehttps://github.com/steed-ict/steedAfter cloning the Steed repository, you can see the following directory:$ls
steed_open_sourceAnd we enter thesteed_open_sourcedirectory and set theSTEED_HOMEenvironment variable:$cdsteed_open_source/
$exportSTEED_HOME=`pwd`3.2 Directory StructureWe usetreecommand to show the directory structure of Steed. You can installtreeusing the following command:$sudoapt-getinstalltreeThe following command shows the directory structure of Steed. Here we usetree . -L 3to show the first three levels of the directory structure:$cd$STEED_HOME$tree.-L2.
├──CMakeLists.txt# CMake configuration file├──CMakeModules# CMake modules│└──FindLZ4.cmake# Find LZ4 module├──LICENSE# open source license├──README.md# README file├──conf# Steed configuration file│└──steed.conf
├──doxygen# Doxygen related files│└──Doxyfile# Doxygen configuration file├──examples# Steed example│├──Makefile# Makefile to build the example│├──json# example json text records││└──try.json│└──main.cpp# example source code├──includea# Steed header files│└──steed.h
├──src# Steed source code│├──CMakeLists.txt│├──steed# Steed core code││├──CMakeLists.txt
││├──assemble# assemble code from columnar to row data││├──base# base source code││├──conf# conf related source code││├──parse# parse json text records to columnar data││├──schema# schema source code││├──steed.cpp# steed main source code││├──store# binary store source code││├──unittest# unittest source code││└──util# util source code│└──thirdparty# thirdparty source code│├──CMakeLists.txt
│├──cli11# CLI11 source code│└──googletest# Google Test source code└──tools├──CMakeLists.txt└──Schema.cpp# schema tool20directories,16files### 3.3 DocumentationSteedusesDoxygentogeneratedocumentation.Youcanusethefollowingcommandtogeneratedocumentation:```bash
$cd$STEED_HOME/doxygen
$doxygenDoxyfileAfter generating documentation, you can opendocs/html/index.htmlto view the documentation.4 Build4.1 Build with CMakeSteed use CMake to build. The following command shows how to build Steed using CMake:$cd$STEED_HOME$mkdirbuild&&cdbuild/&&cmake..The following output shows a successful build:$mkdirbuild&&cdbuild/&&cmake..
--TheCXXcompileridentificationisGNU11.4.0
--DetectingCXXcompilerABIinfo
--DetectingCXXcompilerABIinfo-done--CheckforworkingCXXcompiler:/usr/bin/c++-skipped
--DetectingCXXcompilefeatures
--DetectingCXXcompilefeatures-done--CouldNOTfindPkgConfig(missing:PKG_CONFIG_EXECUTABLE)--CMAKE_MODULE_PATH@/home/steeduser/steed_open_source/CMakeModules
--TheCcompileridentificationisGNU11.4.0
--DetectingCcompilerABIinfo
--DetectingCcompilerABIinfo-done--CheckforworkingCcompiler:/usr/bin/cc-skipped
--DetectingCcompilefeatures
--DetectingCcompilefeatures-done--STEEDdownloadthirdparty@/home/steeduser/steed_open_source/build/thirdparty
--steed@src/steedincludesCLI11lib@/home/steeduser/steed_open_source/src/thirdparty/cli11/include
--CouldNOTfindPython3(missing:Python3_EXECUTABLEInterpreter)--Lookingforpthread.h
--Lookingforpthread.h-found
--PerformingTestCMAKE_HAVE_LIBC_PTHREAD
--PerformingTestCMAKE_HAVE_LIBC_PTHREAD-Success
--FoundThreads:TRUE
--Foundliblz4:
INFOCMAKE_CURRENT_SOURCE_DIR:/home/steeduser/steed_open_source/src/steed
--steedConfigincludesCLI11lib@/home/steeduser/steed_open_source/src/thirdparty/cli11/include
--Foundliblz4-dev:/usr/lib/x86_64-linux-gnu/liblz4.so
--steedincludeslz4lib@/usr/lib/x86_64-linux-gnu/liblz4.so
--Configuringdone--Generatingdone--Buildfileshavebeenwrittento:/home/steeduser/steed_open_source/build4.2 Build with MakeThen you can use the following command to build Steed:$makeallThe following output shows a successful build:$makeall[0%]BuilttargetCLI11[1%]BuildingCXXobjectsrc/thirdparty/googletest/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o[2%]LinkingCXXstaticlibrary../../../../lib/libgtest.a[2%]Builttargetgtest[3%]BuildingCXXobjectsrc/thirdparty/googletest/googlemock/CMakeFiles/gmock.dir/src/gmock-all.cc.o[4%]LinkingCXXstaticlibrary../../../../lib/libgmock.a[4%]Builttargetgmock[5%]BuildingCXXobjectsrc/thirdparty/googletest/googlemock/CMakeFiles/gmock_main.dir/src/gmock_main.cc.o[6%]LinkingCXXstaticlibrary../../../../lib/libgmock_main.a[6%]Builttargetgmock_main[7%]BuildingCXXobjectsrc/thirdparty/googletest/googletest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o[8%]LinkingCXXstaticlibrary../../../../lib/libgtest_main.a[8%]Builttargetgtest_main[9%]BuildingCXXobjectsrc/steed/conf/CMakeFiles/steed_conf.dir/Config.cpp.o[10%]LinkingCXXstaticlibrary../../../lib/libsteed_conf.a[10%]Builttargetsteed_conf[11%]BuildingCXXobjectsrc/steed/util/CMakeFiles/steed_util.dir/Allocator.cpp.o[12%]BuildingCXXobjectsrc/steed/util/CMakeFiles/steed_util.dir/BoolVector.cpp.o[13%]BuildingCXXobjectsrc/steed/util/CMakeFiles/steed_util.dir/Buffer.cpp.o[14%]BuildingCXXobjectsrc/steed/util/CMakeFiles/steed_util.dir/DebugInfo.cpp.o[15%]BuildingCXXobjectsrc/steed/util/CMakeFiles/steed_util.dir/FileHandler.cpp.o[16%]BuildingCXXobjectsrc/steed/util/CMakeFiles/steed_util.dir/FileIO.cpp.o[17%]BuildingCXXobjectsrc/steed/util/CMakeFiles/steed_util.dir/Utility.cpp.o[18%]LinkingCXXstaticlibrary../../../lib/libsteed_util.a[18%]Builttargetsteed_util[20%]BuildingCXXobjectsrc/steed/base/CMakeFiles/steed_base.dir/Block.cpp.o[21%]BuildingCXXobjectsrc/steed/base/CMakeFiles/steed_base.dir/DataType.cpp.o[22%]LinkingCXXstaticlibrary../../../lib/libsteed_base.a[22%]Builttargetsteed_base[23%]BuildingCXXobjectsrc/steed/schema/CMakeFiles/steed_schema.dir/SchemaTree.cpp.o[24%]BuildingCXXobjectsrc/steed/schema/CMakeFiles/steed_schema.dir/SchemaTreeMap.cpp.o[25%]BuildingCXXobjectsrc/steed/schema/CMakeFiles/steed_schema.dir/SchemaTreePrinter.cpp.o[26%]LinkingCXXstaticlibrary../../../lib/libsteed_schema.a[26%]Builttargetsteed_schema[27%]BuildingCXXobjectsrc/steed/store/CMakeFiles/steed_store.dir/CAB.cpp.o[28%]BuildingCXXobjectsrc/steed/store/CMakeFiles/steed_store.dir/CABAppender.cpp.o[29%]BuildingCXXobjectsrc/steed/store/CMakeFiles/steed_store.dir/CABLayouter.cpp.o[30%]BuildingCXXobjectsrc/steed/store/CMakeFiles/steed_store.dir/CABReader.cpp.o[31%]BuildingCXXobjectsrc/steed/store/CMakeFiles/steed_store.dir/CABWriter.cpp.o[32%]BuildingCXXobjectsrc/steed/store/CMakeFiles/steed_store.dir/RecordBuilder.cpp.o[33%]LinkingCXXstaticlibrary../../../lib/libsteed_store.a[33%]Builttargetsteed_store[34%]BuildingCXXobjectsrc/steed/assemble/CMakeFiles/steed_assemble.dir/ColumnAssembler.cpp.o[35%]BuildingCXXobjectsrc/steed/assemble/CMakeFiles/steed_assemble.dir/ColumnExpressionParser.cpp.o[36%]BuildingCXXobjectsrc/steed/assemble/CMakeFiles/steed_assemble.dir/FSMTable.cpp.o[37%]BuildingCXXobjectsrc/steed/assemble/CMakeFiles/steed_assemble.dir/RecordNestedAssembler.cpp.o[38%]BuildingCXXobjectsrc/steed/assemble/CMakeFiles/steed_assemble.dir/RecordOutput.cpp.o[40%]LinkingCXXstaticlibrary../../../lib/libsteed_assemble.a[40%]Builttargetsteed_assemble[41%]BuildingCXXobjectsrc/steed/parse/CMakeFiles/steed_parse.dir/CollectionWriter.cpp.o[42%]BuildingCXXobjectsrc/steed/parse/CMakeFiles/steed_parse.dir/ColumnItemGenerator.cpp.o[43%]BuildingCXXobjectsrc/steed/parse/CMakeFiles/steed_parse.dir/ColumnParser.cpp.o[44%]BuildingCXXobjectsrc/steed/parse/CMakeFiles/steed_parse.dir/JSONRecordBuffer.cpp.o[45%]BuildingCXXobjectsrc/steed/parse/CMakeFiles/steed_parse.dir/JSONRecordNaiveParser.cpp.o[46%]BuildingCXXobjectsrc/steed/parse/CMakeFiles/steed_parse.dir/JSONRecordParser.cpp.o[47%]BuildingCXXobjectsrc/steed/parse/CMakeFiles/steed_parse.dir/JSONTypeMapper.cpp.o[48%]LinkingCXXstaticlibrary../../../lib/libsteed_parse.a[48%]Builttargetsteed_parse[49%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/steed.cpp.o[50%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/conf/Config.cpp.o[51%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/util/Allocator.cpp.o[52%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/util/BoolVector.cpp.o[53%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/util/Buffer.cpp.o[54%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/util/DebugInfo.cpp.o[55%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/util/FileHandler.cpp.o[56%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/util/FileIO.cpp.o[57%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/util/Utility.cpp.o[58%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/base/Block.cpp.o[60%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/base/DataType.cpp.o[61%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/store/CAB.cpp.o[62%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/store/CABAppender.cpp.o[63%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/store/CABLayouter.cpp.o[64%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/store/CABReader.cpp.o[65%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/store/CABWriter.cpp.o[66%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/store/RecordBuilder.cpp.o[67%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/schema/SchemaTree.cpp.o[68%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/schema/SchemaTreeMap.cpp.o[69%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/schema/SchemaTreePrinter.cpp.o[70%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/parse/CollectionWriter.cpp.o[71%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/parse/ColumnItemGenerator.cpp.o[72%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/parse/ColumnParser.cpp.o[73%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/parse/JSONRecordBuffer.cpp.o[74%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/parse/JSONRecordNaiveParser.cpp.o[75%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/parse/JSONRecordParser.cpp.o[76%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/parse/JSONTypeMapper.cpp.o[77%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/assemble/ColumnAssembler.cpp.o[78%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/assemble/ColumnExpressionParser.cpp.o[80%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/assemble/FSMTable.cpp.o[81%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/assemble/RecordNestedAssembler.cpp.o[82%]BuildingCXXobjectsrc/steed/CMakeFiles/steed.dir/assemble/RecordOutput.cpp.o[83%]LinkingCXXsharedlibrary../../lib/libsteed.so[83%]Builttargetsteed[84%]BuildingCXXobjectsrc/steed/unittest/CMakeFiles/test_config.dir/Config.cpp.o[85%]LinkingCXXexecutable../../../bin/test_config[85%]Builttargettest_config[86%]BuildingCXXobjectsrc/steed/unittest/CMakeFiles/test_util.dir/Util.cpp.o[87%]LinkingCXXexecutable../../../bin/test_util[87%]Builttargettest_util[88%]BuildingCXXobjectsrc/steed/unittest/CMakeFiles/test_base.dir/Base.cpp.o[89%]LinkingCXXexecutable../../../bin/test_base[89%]Builttargettest_base[90%]BuildingCXXobjectsrc/steed/unittest/CMakeFiles/test_store.dir/Store.cpp.o[91%]LinkingCXXexecutable../../../bin/test_store[91%]Builttargettest_store[92%]BuildingCXXobjectsrc/steed/unittest/CMakeFiles/test_schema.dir/Schema.cpp.o[93%]LinkingCXXexecutable../../../bin/test_schema[93%]Builttargettest_schema[94%]BuildingCXXobjectsrc/steed/unittest/CMakeFiles/test_parse.dir/Parse.cpp.o[95%]LinkingCXXexecutable../../../bin/test_parse[95%]Builttargettest_parse[96%]BuildingCXXobjectsrc/steed/unittest/CMakeFiles/test_assemble.dir/Assemble.cpp.o[97%]LinkingCXXexecutable../../../bin/test_assemble[97%]Builttargettest_assemble[98%]BuildingCXXobjecttools/CMakeFiles/schema.dir/Schema.cpp.o[100%]LinkingCXXexecutable../bin/schema[100%]Builttargetschema4.3 UnittestDuring build, Steed will build the unittests and tools. The following command shows how to run the unittests:$maketestSteed will run the unittests and show the following output:$maketestRunningtests...
Testproject/home/steeduser/steed_open_source/buildStart1:Test_Config1/7Test#1: Test_Config ......................   Passed    0.00 secStart2:Test_Util2/7Test#2: Test_Util ........................   Passed    0.01 secStart3:Test_Base3/7Test#3: Test_Base ........................   Passed    0.00 secStart4:Test_Store4/7Test#4: Test_Store .......................   Passed    0.00 secStart5:Test_Schema5/7Test#5: Test_Schema ......................   Passed    0.00 secStart6:Test_Parse6/7Test#6: Test_Parse .......................   Passed    0.00 secStart7:Test_Assemble7/7Test#7: Test_Assemble ....................   Passed    0.00 sec100%testspassed,0testsfailedoutof7TotalTesttime(real)=0.02secAll the unittests are passed.The executable files of the unittests and tools are in thebuild/bindirectory.$tree$STEED_HOME/build/bin
/home/steeduser/steed_open_source/build/bin/
├──schema# schema tool├──test_assemble# assemble unittest├──test_base# base unittest├──test_config# config unittest├──test_parse# parse unittest├──test_schema# schema unittest├──test_store# store unittest└──test_util# util unittest0directories,8filesYou can also run the executable files in thebuild/bindirectory directly.4.4 InstallAfter build, you can install the steed library and executable file using the following command:$sudomakeinstallThe following output shows a successful install:$sudomakeinstall[0%]BuilttargetCLI11
Consolidatecompilergenerateddependenciesoftargetgtest[2%]Builttargetgtest
Consolidatecompilergenerateddependenciesoftargetgmock[4%]Builttargetgmock
Consolidatecompilergenerateddependenciesoftargetgmock_main[6%]Builttargetgmock_main
Consolidatecompilergenerateddependenciesoftargetgtest_main[8%]Builttargetgtest_main
Consolidatecompilergenerateddependenciesoftargetsteed_conf[10%]Builttargetsteed_conf
Consolidatecompilergenerateddependenciesoftargetsteed_util[18%]Builttargetsteed_util
Consolidatecompilergenerateddependenciesoftargetsteed_base[22%]Builttargetsteed_base
Consolidatecompilergenerateddependenciesoftargetsteed_schema[26%]Builttargetsteed_schema
Consolidatecompilergenerateddependenciesoftargetsteed_store[33%]Builttargetsteed_store
Consolidatecompilergenerateddependenciesoftargetsteed_assemble[40%]Builttargetsteed_assemble
Consolidatecompilergenerateddependenciesoftargetsteed_parse[48%]Builttargetsteed_parse
Consolidatecompilergenerateddependenciesoftargetsteed[83%]Builttargetsteed
Consolidatecompilergenerateddependenciesoftargettest_config[85%]Builttargettest_config
Consolidatecompilergenerateddependenciesoftargettest_util[87%]Builttargettest_util
Consolidatecompilergenerateddependenciesoftargettest_base[89%]Builttargettest_base
Consolidatecompilergenerateddependenciesoftargettest_store[91%]Builttargettest_store
Consolidatecompilergenerateddependenciesoftargettest_schema[93%]Builttargettest_schema
Consolidatecompilergenerateddependenciesoftargettest_parse[95%]Builttargettest_parse
Consolidatecompilergenerateddependenciesoftargettest_assemble[97%]Builttargettest_assemble
Consolidatecompilergenerateddependenciesoftargetschema[100%]Builttargetschema
Installtheproject...
--Installconfiguration:"release"--Up-to-date:/usr/local/include
--Installing:/usr/local/include/gmock
--Installing:/usr/local/include/gmock/internal
--Installing:/usr/local/include/gmock/internal/gmock-port.h
--Installing:/usr/local/include/gmock/internal/gmock-internal-utils.h
--Installing:/usr/local/include/gmock/internal/custom
--Installing:/usr/local/include/gmock/internal/custom/README.md
--Installing:/usr/local/include/gmock/internal/custom/gmock-port.h
--Installing:/usr/local/include/gmock/internal/custom/gmock-generated-actions.h
--Installing:/usr/local/include/gmock/internal/custom/gmock-matchers.h
--Installing:/usr/local/include/gmock/internal/gmock-pp.h
--Installing:/usr/local/include/gmock/gmock-spec-builders.h
--Installing:/usr/local/include/gmock/gmock-cardinalities.h
--Installing:/usr/local/include/gmock/gmock-more-actions.h
--Installing:/usr/local/include/gmock/gmock.h
--Installing:/usr/local/include/gmock/gmock-more-matchers.h
--Installing:/usr/local/include/gmock/gmock-actions.h
--Installing:/usr/local/include/gmock/gmock-function-mocker.h
--Installing:/usr/local/include/gmock/gmock-matchers.h
--Installing:/usr/local/include/gmock/gmock-nice-strict.h
--Installing:/usr/local/lib/libgmock.a
--Installing:/usr/local/lib/libgmock_main.a
--Installing:/usr/local/lib/pkgconfig/gmock.pc
--Installing:/usr/local/lib/pkgconfig/gmock_main.pc
--Installing:/usr/local/lib/cmake/GTest/GTestTargets.cmake
--Installing:/usr/local/lib/cmake/GTest/GTestTargets-release.cmake
--Installing:/usr/local/lib/cmake/GTest/GTestConfigVersion.cmake
--Installing:/usr/local/lib/cmake/GTest/GTestConfig.cmake
--Up-to-date:/usr/local/include
--Installing:/usr/local/include/gtest
--Installing:/usr/local/include/gtest/internal
--Installing:/usr/local/include/gtest/internal/gtest-string.h
--Installing:/usr/local/include/gtest/internal/gtest-port.h
--Installing:/usr/local/include/gtest/internal/custom
--Installing:/usr/local/include/gtest/internal/custom/README.md
--Installing:/usr/local/include/gtest/internal/custom/gtest-port.h
--Installing:/usr/local/include/gtest/internal/custom/gtest-printers.h
--Installing:/usr/local/include/gtest/internal/custom/gtest.h
--Installing:/usr/local/include/gtest/internal/gtest-type-util.h
--Installing:/usr/local/include/gtest/internal/gtest-internal.h
--Installing:/usr/local/include/gtest/internal/gtest-param-util.h
--Installing:/usr/local/include/gtest/internal/gtest-filepath.h
--Installing:/usr/local/include/gtest/internal/gtest-port-arch.h
--Installing:/usr/local/include/gtest/internal/gtest-death-test-internal.h
--Installing:/usr/local/include/gtest/gtest-assertion-result.h
--Installing:/usr/local/include/gtest/gtest-message.h
--Installing:/usr/local/include/gtest/gtest-param-test.h
--Installing:/usr/local/include/gtest/gtest-matchers.h
--Installing:/usr/local/include/gtest/gtest_pred_impl.h
--Installing:/usr/local/include/gtest/gtest-spi.h
--Installing:/usr/local/include/gtest/gtest-typed-test.h
--Installing:/usr/local/include/gtest/gtest-death-test.h
--Installing:/usr/local/include/gtest/gtest-printers.h
--Installing:/usr/local/include/gtest/gtest-test-part.h
--Installing:/usr/local/include/gtest/gtest.h
--Installing:/usr/local/include/gtest/gtest_prod.h
--Installing:/usr/local/lib/libgtest.a
--Installing:/usr/local/lib/libgtest_main.a
--Installing:/usr/local/lib/pkgconfig/gtest.pc
--Installing:/usr/local/lib/pkgconfig/gtest_main.pc
--Installing:/usr/local/lib/libsteed.so
--Installing:/usr/local/include/steed.hInCMakeLists.txt, we define the install directory using the parameterCMAKE_INSTALL_PREFIX. The default install directory is/usr/local/bin. You can also change the install directory using the following command:$cmake-DCMAKE_INSTALL_PREFIX=/mydir..or you can change the install directory in theCMakeLists.txtfile:set(CMAKE_INSTALL_PREFIX/mydir)Then you can find the following files in the install directory, where${CMAKE_INSTALL_PREFIX}is our default install directory/usr/local:$tree/usr/local/-L2/usr/local/
├──# other files├──include
│└──steed.h# steed header file└──lib├──libSteed.so# STEED shared library└──# other fileslibSteed.sois the steed shared library. You can use the steed shared library to build your own application. We will show you how to build your own application with the steed shared library in the next section.5 ExampleIn this section, we will show you how to build your own application with the steed shared library. We will show you how to build the example application in theexamplesdirectory. The directory structure of the example is as follows:$cd$STEED_HOME/examples
$tree
.
├──Makefile
├──json
│└──try.json# example json text data└──main.cpp# example source codeIn theexamplesdirectory, we have the following files:Makefileis the Makefile to build the example application;jsondirectory contains the example json text data;main.cppis the example source code.5.1 Build with Steed shared libraryWe use the following Makefile to build the example application:# steed examplePROJECT=SteedExampleLIBSTEED=steedBINARY=$(PROJECT)THIRDDIRS=../src/thirdparty/cli11/includeSRCSUBDIR+=${THIRDDIRS}SRCSUBDIR+=../src/steed/confSRCSUBDIR+=../src/steed/utilSRCSUBDIR+=../src/steed/baseSRCSUBDIR+=../src/steed/schemaSRCSUBDIR+=../src/steed/storeSRCSUBDIR+=../src/steed/assembleSRCSUBDIR+=../src/steed/parseSRCSUBDIR+=../src/steed/unittestINCLUDES=$(addprefix-I,$(SRCSUBDIR))# compilerCC=g++CFLAGS=-std=c++11-Wall-Wextra-Werror-WallCFLAGS+=-g-O3-fPICCFLAGS+=-I../include$(INCLUDES)# linkerLDFLAGS=-ldl-L/usr/local/lib-l$(LIBSTEED)# source filesSRCS=main.cpp# executable file$(BINARY):$(SRCS)$(CC)-o$@$^$(CFLAGS)$(LDFLAGS)# cleanclean:rm-rf$(BINARY)$(OBJS)In the Makefile, we use theLDFLAGSvariable to link the steed shared library. The following is the output of themakecommand:$make
g++-oSteedExamplemain.cpp-std=c++11-Wall-Wextra-Werror-Wall-g-O3-fPIC-I../include-I../src/thirdparty/cli11/include-I../src/steed/conf-I../src/steed/util-I../src/steed/base-I../src/steed/schema-I../src/steed/store-I../src/steed/assemble-I../src/steed/parse-I../src/steed/unittest-ldl-L/usr/local/lib-lsteed5.2 JSON Example DataIn theexamples/json/try.jsonfile, we have the following json records as an example:{"item":"journal","status":"A","size":{"h":14,"w":21,"uom":"cm"},"instock":[{"warehouse":"A","qty":5}]}{"item":"postcard","status":"A","size":{"h":10,"w":15.25,"uom":"cm"},"instock":[{"warehouse":"B","qty":15},{"warehouse":"C","qty":35}]}{"item":"journal","qty":25,"tags":["blank","red"],"size":{"h":14,"w":21,"uom":"cm"}}{"item":"mat","qty":85,"tags":["gray"],"size":{"h":27.9,"w":35.5,"uom":"cm"}}{"item":"mousepad","qty":25,"tags":["gel","blue"],"size":{"h":19,"w":22.85,"uom":"cm"}}In json records, we have the following fields:"item" is a string field and appears in all records;"status" is a string field and appears in the first two records;"size" is a nested object field and appears in all records. It has three subfields, including "h", "w" and "uom";"instock" is a nested array field and appears in the first two records. It has two subfields, including "warehouse" and "qty";"qty" is a number field and appears in the last three records;"tags" is a string array field and appears in the last three records.
Then we can use the json records as an example to show how to parse json data to columnar data and assemble columnar data to row data.5.3 Steed ConfigurationSteed uses CLI11 to parse the configuration file. The default configuration file is./conf/steed.conf. The following is an example of the configuration file:### Binary Storage Settings# store base dir:#   the base dir of all column and schema datastore_base=./data# column data dir:#   the dir used to store column datadata_dir=cols# schema dir:#   the dir used to store schema dataschema_dir=schema### Steed Runtime Settings# mem align size:#   the size of memory alignmem_align_size=4096# max column number:#   the record capacity of a CAB (Column Aligned Block)cab_recd_num=8# max record number:#   the record capacity in batch during parsing text json recordstext_recd_num=16# record number in a text fileIn this configuration file, we set the store configuration as follows:optionstore_baseset the base directory as./data;optiondata_dirset the column data directory name ascols;optionschema_dirset the schema directory name asschema.We also set the runtime configuration as follows:optionmem_align_sizeset the memory alignment size as 4096;optioncab_recd_numset the record number in a CAB file as 8, which means that there are 8 records in a CAB;optiontext_recd_numset the record number in a text file as 16, which means steed will parse 16 records in a batch during parsing JSON records.5.4 Run ExampleIn the main.cpp file, we use the steed library to parse the json data in thejsondirectory to columnar data. The following is themain.cppfile:#include"steed.h"usingnamespacesteed;intmain(intargc,char*argv[]){(void)argc,(void)argv;// use default config filestringconf_file("");steed::init(conf_file);std::stringdb("demo"),table("mytry");createDatabase(db);createTable(db,table);{// create a column parser to parse JSON recordsstd::stringjpath("./json/try.json");ifstreamifs(jpath);if(!ifs.is_open()){printf("Main: cannot open [%s]!\n",jpath.c_str());return-1;}// ifssteed::ColumnParser*cp=newsteed::ColumnParser();istream*is=&ifs;if(cp->init(db,table,is)<0){printf("Main: ColumnParser init failed!\n");return-1;}// ifintstatus=0;while((status=cp->parseOne())>0){/* do nothing */}deletecp;cp=nullptr;ifs.close();if(status<0){printf("Main: insert failed!\n");return-1;}// if}{// create a column assembler to assemble binary records and outputstd::vector<std::string>cols;cols.emplace_back("item");cols.emplace_back("size.h");ColumnAssembler*ca=newColumnAssembler();if(ca->init(db,table,cols)<0){printf("STEED::init ColumnAssembler failed!\n");return-1;}// ifchar*rbgn=nullptr;ostream*ostrm=&std::cout;RecordOutputro(ca->getSchemaTree());while(ca->getNext(rbgn)>0){ro.outJSON2Strm(ostrm,rbgn);}deleteca;ca=nullptr;}dropTable(db,table);dropDatabase(db);steed::uninit();return0;}// mainThe example has the following steps:In the main function, we first use theinitfunction to initialize the steed library.Then, we create a database nameddemoand a table namedmytryin thedemodatabase.Then the example creates an ifstream object to read the json data in thejsondirectory and use this object to construct aColumnParserobject. Then it uses the ColumnParser object to parse the json data to columnar data.After parsing the json data, the example creates aColumnAssemblerobject to assemble the columnar data to row data and output the row data to the standard output.Finally, the example drops themytrytable anddemodatabase.5.4.1 Config and init StaticSteer uses theConfigclass to store the configuration. We define a globalConfigobject namedg_configinsrc/steed/steed.cppand use theinitanduninitfunction to initialize and uninit the steed static data. Here we use an empty string to init and use the default config file./conf/steed.confto initialize the steed library:intmain(intargc,char*argv[]){(void)argc,(void)argv;// use default config filestringconf_file("");steed::init(conf_file);// ...steed::uninit();return0;}// main5.4.2 Create DatabaseIn this section, we will show you how to create a database. We set the store base directory as./datain the configuration file and the directory is empty now.You can create thedatadirectory manually or steed will create thedatadirectory automatically when you create a database. Now thedatadirectory is empty:$tree./data
.0directories,0filesIn the main function, we use thecreateDatabasefunction to create a database nameddemo:intmain(intargc,char*argv[]){// ...std::stringdb("demo"),table("mytry");createDatabase(db);// ...}// mainSteed will create thedemodatabase directory in the store base directory. There are two subdirectories in the database directory, includingcolsandschema. Thecolsdirectory is used to store the columnar data, theschemadirectory is used to store the schema of the table.$tree./data
./data
└──demo├──cols└──schema3directories,0files5.4.3 Create TableThen we can create a table namedmytryin thedemodatabase in the main function:intmain(intargc,char*argv[]){// ...std::stringdb("demo"),table("mytry");createDatabase(db);createTable(db,table);// ...}// mainAfter that, we can find themytrydirectory in thecolsandschemadirectories. themytrydirectory in theschemadirectory is used to store the schema of themytrytable, and themytrydirectory in thecolsdirectory is used to store the columnar data of themytrytable. Since there is no data in themytrytable, themytrydirectory in thecolsdirectory is empty now.$tree./data
./data
└──demo├──cols│└──mytry└──schema└──mytry5.4.4 Parse DataNow we can parse the json data in thejsondirectory to columnar data. In the main function, we use theparseOnefunction to parse the json records to columnar data one by one. Steed also provides theparseAllfunction to parse all the json records in a batch. Here is an example:intmain(intargc,char*argv[]){// ...{// create a column parser to parse JSON recordsstd::stringjpath("./json/try.json");ifstreamifs(jpath);if(!ifs.is_open()){printf("Main: cannot open [%s]!\n",jpath.c_str());return-1;}// ifssteed::ColumnParser*cp=newsteed::ColumnParser();istream*is=&ifs;if(cp->init(db,table,is)<0){printf("Main: ColumnParser init failed!\n");return-1;}// ifintstatus=0;while((status=cp->parseOne())>0){/* do nothing */}deletecp;cp=nullptr;ifs.close();if(status<0){printf("Main: insert failed!\n");return-1;}// if}// ...}// mainSteed use theColumnParserclass to parse the json data to columnar data. TheColumnParserclass provides theinitfunction to initialize theColumnParserobject. The following is theinitfunction in ColumnParser.h:/*** init to parse: infer the SchemaTree from SampleTree* @param db   database name* @param clt  table name* @param is   JSON text records input stream* @return 0 success; <0 failed*/intinit(conststring&db,conststring&clt,istream*is);You could also create a std::istringstream object to read the json data from a string. Here is an example:std::stringjstr="{\"item\":\"journal\",\"status\":\"A\",\"size\": {\"h\": 14,\"w\": 21,\"uom\":\"cm\"},\"instock\": [ {\"warehouse\":\"A\",\"qty\": 5 } ] }";std::istringstreamiss(jstr);ColumnParsercp("demo","mytry",&iss);Then the ColumnParser object will parse the json data to columnar data and store the columnar data in themytrydirectory under thecolsdirectory.After that, we can find the columnar data in themytrydirectory in thecolsdirectory. Themytrydirectory contains the columnar data of themytrytable. Themytrydirectory contains the following files:$tree./data
./data
└──demo├──cols│└──mytry│├──item#1#I.cab│├──item#1#I.cab.info│├──size#1#A.h#1#H.cab│├──size#1#A.h#1#H.cab.info│├──tags#2#I.cab│├──tags#2#I.cab.info│└──# other columnar data└──schema└──mytryAfter parsing, steed generates 2 files for each column:.cabfor the columnar data (aligned by ColumnAlignedBlock, CAB for short) and.cab.infofor the CAB header information.For example,size#1#A.h#1#H.cabis the CAB file for thesize.hcolumn. We use path expression to express the nested field: thesizecolumn is a nested object column, and thehsubfield is a number column separated by '.'. Thesize#1#A.h#1#H.cabfile is the CAB file for thehsubfield.Steed also update the schema info during the parsing progress. After that, the schema info will be flushed to the filedemo/schema/mytry.5.4.5 Assemble DataUsing theassemblefunction, we can assemble the columnar data to row data. In the main function, we use theassemblefunction to assemble the columnar data to row data:intmain(intargc,char*argv[]){// ...{// create a column assembler to assemble binary records and outputstd::vector<std::string>cols;cols.emplace_back("item");cols.emplace_back("size.h");ColumnAssembler*ca=newColumnAssembler();if(ca->init(db,table,cols)<0){printf("STEED::init ColumnAssembler failed!\n");return-1;}// ifchar*rbgn=nullptr;ostream*ostrm=&std::cout;RecordOutputro(ca->getSchemaTree());while(ca->getNext(rbgn)>0){ro.outJSON2Strm(ostrm,rbgn);}deleteca;ca=nullptr;}// ...}// mainIn this example, we only assemble theitemandsize.hcolumns. Steed will assemble theitemandsize.hcolumns to row data and print the row data to the standard output. We create a ColumnAssembler object to assemble the columnar data to row data. It is inited by theinitfunction:/*** init to assemble: infer the SchemaTree from SampleTree* @param db   database name* @param clt  table name* @param cols column names to assemble* @return 0 success; <0 failed*/intinit(conststring&db,conststring&clt,constvector<string>&cols);Then the ColumnAssemble object will assemble the columnar data to row data and RecordOutput object will output the row data to the standard output.  The following is the row data:{"item":"journal","size":{"h":14.000000}}{"item":"postcard","size":{"h":10.000000}}{"item":"journal","size":{"h":14.000000}}{"item":"mat","size":{"h":27.900000}}{"item":"mousepad","size":{"h":19.000000}}5.4.6 Drop Table and DatabaseFinally, we can drop the table and database using thedropTableanddropDatabasefunctions. In the main function, we use thedropTableanddropDatabasefunctions to drop themytrytable anddemodatabase:intmain(intargc,char*argv[]){// ...dropTable(db,table);dropDatabase(db);// ...}// mainThen we can find thedemodirectory in the store base directory is removed.$tree./data
./data0directories,0files