Package: thoughtspot-tml
ThoughtSpot TMLa Python package for working withThoughtSpotModeling Language (TML) files programmaticallyInstallation|Example|Migration to v2.0.0|Reference|Notes🚨If your examples or scripts are built onthoughtspot_tml==1.3.0, see ourMigration to v2.0.0guide. 🚨FeaturesSupports: Connections,Tables,Views,SQLViews,Worksheets,Answers,LiveboardsDeep attribute accessRoundtripping from text or fileUtilities fordisambiguationworkflowsThis package will not perform validation of the constructed TML files or interact with yourThoughtSpotcluster!Please leverage theThoughtSpotREST APIfor this purpose.Installationthoughtspot_tmlrequires at leastPython 3.7,preferablyPython 3.9and above.Installation is as simple as:pipinstallthoughtspot-tmlA Basic ExampleThis example creates a command-line tool for changing the prefix in the names of the Table objects that a Worksheet object connects to.# worksheet_remapping.pyfromthoughtspot_tmlimportWorksheetimportargparseimportpathlibdeffilepath(fp:str)->pathlib.Path:"""Converts a string to a pathlib.Path."""path=pathlib.Path(fp)ifnotpath.exists():raiseargparse.ArgumentTypeError(f"path '{fp!r}' does not exist")ifnotpath.is_file():raiseargparse.ArgumentValueError(f"path must be a file, got '{fp!r}'")returnpathdefmain():# Create a command line application#   - argument for a WORKSHEET.worksheet.tml#   - options for the "before" and "after" tabling naming conventionsparser=argparse.ArgumentParser()parser.add_argument("worksheet_tml",help="a worksheet.tml to remap",type=filepath)parser.add_argument("-s","--src-prefix",metavar="SRC",default="DEV_",type=str,help="(default:%(default)s)")parser.add_argument("-d","--dst-prefix",metavar="DST",default="TEST_",type=str,help="(default:%(default)s)")# Parse CLI inputargs=parser.parse_args()# Read from filetml=Worksheet.load(args.worksheet_tml)# Replace instances of DEV_ with TEST_fortableintml.worksheet.tables:table.name=table.name.replace(args.src_prefix,args.dst_prefix)# Save to filetml.dump(args.worksheet_tml)if__name__=='__main__':raiseSystemExit(main())>>>pythonworksheet_remapping.py-h

usage:[-h][-sSRC][-dDST]worksheet_tml

positionalarguments:worksheet_tmlaworksheet.tmltoremap

options:-h,--helpshowthishelpmessageandexit-sSRC,--src-prefixSRC(default:DEV_)-dDST,--dst-prefixDST(default:TEST_)A more complex version of this example, as well as more examples can be found in the/examples directoryin this repository.thoughtspot_tmlReferenceTML Objects|Deserialization|Serialization|SpotApp|UtilitiesTML Objectsfromthoughtspot_tmlimportTable,View,SQLView,Worksheetfromthoughtspot_tmlimportAnswer,Liveboard,Pinboard# aliasesfromthoughtspot_tmlimportThoughtSpotView# Viewfromthoughtspot_tmlimportSavedAnswer# Answerfromthoughtspot_tmlimportSystemTable# TableEach TML object has a top-level attribute for the globally unique identifier, orGUID, as well as the document form of the object it represents. This identically mirrors the TML specification you can findin theThoughtSpotdocumentation. In addition, thenameattribute of the TML document itself has been pulled into the top-level namespace.@dataclassclassWorksheet(TML):"""Representation of a ThoughtSpot Worksheet TML."""guid:GUIDworksheet:WorksheetEDocProto@propertydefname(self)->str:returnself.worksheet.nameThe full, composable TML specification can found in_scriptability.py. Each piece of the spec is a pythondataclasses.dataclassfield. The internal_scriptability.pymodule is generated code from theThoughtSpot's internal architecture and allows forthoughtspot_tmlto offer the deep attribute access experience in python.@dataclassclassTable(TML):"""Representation of a ThoughtSpot Table TML."""guid:GUIDtable:LogicalTableEDocProto@propertydefname(self)->str:returnself.table.nameFor example, interesting attributes about the Table TML spec are exposed via attributes which can, in turn expose their own attributes themselves. This functionality offers common pattersn to be expressed natively in Python, such as remapping a Table's connection details.tml=Table.load("tests/data/DUMMY.table.tml")# get the Table document objecttml.table# => LogicalTableEdocProto(...)# get the Table's underlying connected detailstml.table.db# => 'PMMDB'tml.table.schema# => 'RETAILAPPAREL'tml.table.db_table# => 'dim_retapp_products'# get the Table's columnstml.table.columns# => [LogicalTableEDocProtoLogicalColumnEDocProto(...), ...]# repoint this ThoughtSpot Table to a new external tabletml.table.schema="RETAILAPPAREL_V2"tml.table.db_table="DIM_RETAPP_PRODUCTS"TheConnectionis a special type of TML object. Connections (also known as "Embrace" Connections) were implemented prior to the TML spec being officially released. The remapping file (connection.yaml), obtained from your platform atData > Connections > (...) in the top right > Remapping > Downloaddefines howThoughtSpottable objects relate to their external counterparts.fromthoughtspot_tmlimportConnection# aliasesfromthoughtspot_tmlimportEmbraceConnection# ConnectionEven though the resulting file is different, we've implemented the Connection with an identical form.The Connection GUID, while optional inthoughtspot_tml, is required when modifying or removing an existing connection via the REST API. A Connection's GUID can be obtained by calling theconnection/listendpoint.When loading from a file, ifthoughtspot_tmlidentifies the filename is a GUID, then the property will be set on the resulting object.Theconnection/updateREST API endpoint requires connections to formatted in a different way. For this, we provide a method to generate the metadata parameter data, which is a mapping of configuration attributes, as well as database, schema, and table objects.*ThoughtSpotplans to release Connection TML in a future release.@dataclassclassConnection(TML):"""Representation of a ThoughtSpot Connection YAML."""guid:Optional[GUID]connection:ConnectionDocdefto_rest_api_v1_metadata(self)->ConnectionMetadata:...Each object contains multiple methods for serialization and deserialization.DeserializationFor deserialization of a TML document into a python object.ws=Worksheet.load(path:PathLike="tests/data/DUMMY.worksheet.tml")ws=Worksheet.loads(tml_document:str=...)# can be obtained from the ThoughtSpot REST APIws.guid=="2ea7add9-0ccb-4ac1-90bb-231794ebb377".loada worksheet from a.worksheet.tmlfile, or as a string directly from themetadata/tml/exportAPI with.loads.SerializationFor serialization of a TML python object back into data.data=ws.to_dict()data["guid"]=="2ea7add9-0ccb-4ac1-90bb-231794ebb377"ws.dump(path="tests/data/DUMMY.worksheet.tml")# DUMMY.worksheet.tml## guid: 2ea7add9-0ccb-4ac1-90bb-231794ebb377# worksheet:#   ...data_s=ws.dumps(format_type="YAML")data=yaml.load(data_s)data["guid"]=="2ea7add9-0ccb-4ac1-90bb-231794ebb377"# -or-data=ws.dumps(format_type="JSON")data_s=json.loads(data_s)data["guid"]=="2ea7add9-0ccb-4ac1-90bb-231794ebb377".to_dictto convert the entire object tree into python native types, or write back to a file with.dumpas a TML-formatted string. The formatting can be overriden to JSON if the JSON file type is used (.worksheet.json)..dumpsallows access to the formatted string directly, typically used as input for themetadata/tml/importAPI.SpotAppfromthoughtspot_tmlimportSpotAppSpotApps are bundles of TML which can be obtained directly from theThoughtSpotuser interace as a zip file archive or from the/metadata/tml/exportAPI endpoint using theexport_associated = truequery parameter.export_response=...# /metadata/tml/exports=SpotApp.from_api(export_response)print(s.tml)# => [Worksheet(...), Table(...), Table(...)]print(s.manifest)# => Manifest(...)# -or-s=SpotApp.read("tests/data/DUMMY_spot_app.zip")print(s.tml)# => [Worksheet(...), Table(...), Table(...)]print(s.manifest)# => Manifest(...)SpotApps can also be saved to a new zipfile archive through the.savemethod.s=SpotApp.read("tests/data/DUMMY_spot_app.zip")s.save("tests/data/NEW_DUMMY_spot_app.zip")Utilitiesdetermine_tml_type|EnvironmentGUIDMapper|disambiguatethoughtspot_tml.utilsare additional methods which can help or speed up working with TML documents.determine_tml_typeTML is both a data structure and file format, and these formats vary slightly across each document.determine_tml_typewill return the appropriate TML class so that you can calldeserialization methodsdirectly. Pass either thepathkeyword with a filepath, or the file info directly from one of the objects returned in the/metadata/tml/exportresponse data.signaturedefdetermine_tml_type(*,info:TMLDocInfo=None,path:PathLike=None)->Union[Connection,TMLObject]:"""Get the appropriate TML class based on input data.Parameters----------info : TMLDocInfoAPI edoc info responsepath : PathLikefilepath to parseRaises------TMLError, when a valid TML type could not be found based on input"""usagefromthoughtspot_tml.utilsimportdetermine_tml_typetml_cls=determine_tml_type(path="/tests/data/DUMMY.worksheet.tml")tml=tml_cls.load(path="/tests/data/DUMMY.worksheet.tml")type(tml)isWorksheet# => True# -or-export_response=...# /metadata/tml/exporttml_cls=determine_tml_type(info=export_response["object"][0]["info"])tml=tml_cls.loads(tml_document=export_response["object"][0]["edoc"])type(tml)isWorksheet# => TrueEnvironmentGUIDMapperTheEnvironmentGUIDMapperis a dictionary-like data structure which can help you maintain references to objects across yourThoughtSpotenvironments. The underlying data structure is intended to clearly show the relationship of a given object between any number of environments. An "environment" can be any scope you consider separate from each other, be it 2ThoughtSpotservers, 2 Connections on the same server, or even "Copy of" the same object within a single Connection.signatureclassEnvironmentGUIDMapper:"""Attributes----------environment_transformer : Callable(str) -> stra function which transforms the ENV name before adding it to the mapping"""def__init__(self,environment_transformer:Callable[[str],str]=str.upper):usagefromthoughtspot_tml.utilsimportEnvironmentGUIDMapper# create a new mappermapper=EnvironmentGUIDMapper()# or EnvironmentGUIDMapper.read(path=...)# map 3 guids to represent the same ThoughtSpot object across environmentsmapper["guid1"]=("PROD","guid1")# 1. add a new guid into the mappermapper["guid1"]=("TEST","guid2")# 2. map guid1 to a guid in another environmentmapper["guid2"]=("DEV","guid3")# 3. map a new guid3 to any of existing guid# persist the mapping file to diskmapper.save(path="marketing_thoughtspot_guid_mapping.json")# what's the JSON data structure look like?print(mapper){"guid1__guid2__guid3":{"PROD":"guid1","TEST":"guid2","DEV":"guid3"}}# create a new mapper from a filenew_mapper=EnvironmentGUIDMapper.read(path="marketing_thoughtspot_guid_mapping.json")# add another object mappingnew_mapper.set("guid10",environment="PROD",guid="guid10")# equivalent to new_mapper["guid10"] = ("PROD", "guid10")new_mapper.set("guid10",environment="TEST",guid="guid11")new_mapper.set("guid10",environment="DEV",guid="guid12")# get all the environments that would map to "guid10"print(new_mapper["guid10"])# or  new_mapper.get("guid10"){"PROD":"guid10","TEST":"guid11","DEV":"guid12"}# get a mapping of all DEV -> PROD related ThoughtSpot objectsprint(new_mapper.generate_mapping(from_environment="DEV",to_environment="PROD")){"guid3":"guid1","guid12":"guid10"}disambiguateInThoughtSpot, the uniqueness constraint exists on the underlying object'sguid. This means that there can be multiple objects of the same type with the same name. An example of this is maintaining both a DEV and PROD Connection. All the development work happens on one set of objects (that are not shared with any of the End User community), while the production connection contains objects with identical names thatareshared with the End User community.To reduce ambiguity, you may need to add thefqnkey to your TML document when you reference source tables or connections. If you do not add thefqnkey, and the connection or table you reference does not have a unique name, the import will fail.NOTE:Prior toThoughtSpotV8.7.0, TML does not export with thefqnautomatically.signaturedefdisambiguate(tml:TMLObject,*,guid_mapping:Dict[str,GUID],remap_object_guid:bool=True,delete_unmapped_guids:bool=False,)->TMLObject:"""Deep scan the TML looking for fields to add FQNs to.This will explore the top-level guid and all nested objects looking onTables, Worksheets, etc to disambiguate.Parameters----------tml : TMLObjectthe tml to scanguid_mapping : {str: GUID}a mapping of names or guids, to the FQN to add to the objectremap_object_guid : bool = Truewhether or not to remap the tml.guiddelete_unmapped_guids : bool = Falseif a match could not be found, set the FQN and object guid to None"""usagefromthoughtspot_tml.utilsimportdisambiguatefromthoughtspot_tmlimportWorksheet# Load a Worksheet and check its dataws=Worksheet.load("tests/data/DUMMY.worksheet.tml")ws.guid=="2ea7add9-0ccb-4ac1-90bb-231794ebb377"# => Truews.worksheet.tables[0].name=="dim_retapp_products"# => Truews.worksheet.tables[0].fqnisNone# => True# Assign a Table an FQN. This information can be retrieved from ThoughtSpot REST API metadata/list.ws=disambiguate(ws,guid_mapping={"dim_retapp_products":"7fd39fdb-9dfe-4954-b5dd-9a5d846085b0"})ws.worksheet.tables[0].fqnisNone# => Falsews.worksheet.tables[0].fqn=="7fd39fdb-9dfe-4954-b5dd-9a5d846085b0"# => True# Re-assign the GUID to a new environment.ws=disambiguate(ws,guid_mapping={"7fd39fdb-9dfe-4954-b5dd-9a5d846085b0":"99999999-9999-4999-9999-999999999999"})ws.worksheet.tables[0].fqn=="7fd39fdb-9dfe-4954-b5dd-9a5d846085b0"# => Truews.worksheet.tables[0].fqn=="99999999-9999-4999-9999-999999999999"# => False# Remove GUIDs which aren't found in the mapping, including the top-level GUID.ws=disambiguate(ws,guid_mapping={},delete_unmapped_guids=True)ws.worksheet.tables[0].name=="dim_retapp_products"# => Truews.worksheet.tables[0].fqnisNone# => Truews.guidisNone# => TrueThedisambiguatefunction will walk through thethoughtspot_tmlTML object specifying the.fqnbased on keys in theguid_mappingdictionary.Theguid_mappingwill typically be a mapping of GUIDs between 2 environments, but the "before" environment can be any string. This can be helpful to quickly addfqnto any object which has yet to define it.Theremap_object_guid(default:True) will consider the top-levelTML.guidas a candidate for re-mapping.Thedelete_unmapped_guids(default:False) will remove any.fqns which are not found in theguid_mapping.Migration to v2.0.0WithV2.0.0, we now programmatically build the TML spec from the underlying microservice's data structure. The largest benefit of this move is that we can nowRound-tripping to FileThe utility classYAMLTMLhas been replaced withutils.determine_tml_typeand a private base classTML, which all public metadata objects inherit from. The TML type which is returned has the appropriate [de]serialization methods.Both of the following patterns represent round-tripping.importpathlibworksheet_fp="tests/data/DUMMY.worksheet.tml"worksheet_tml_str=pathlib.Path(worksheet_fp).read_text()# V1.3.0fromthoughtspot_tmlimportYAMLTMLtml=YAMLTML.get_tml_object(worksheet_tml_str)tml_document_str=YAMLTML.dump_tml_object(tml)# V2.0.0fromthoughtspot_tml.utilsimportdetermine_tml_typefromthoughtspot_tmlimportWorksheettml_cls=determine_tml_type(path=worksheet_fp)tml=tml_cls.loads(worksheet_tml_str)# any one of these methods..# tml = tml_cls.load(worksheet_fp)# tml = Worksheet.loads(worksheet_tml_str)# tml = Worksheet.load(worksheet_fp)tml_document_str=tml.dumps(worksheet_fp)Identifying the TML Object TypeTo identify the type of TML object you are working with inV1.3.0you would use.content_type, withV2.0.0you can now use.tml_type_name.GUID & FQN HandlingInV1.3.0, GUIDs were deleted from the underlying data structure with.remove_guid()in order to ensure the REST API created new objects. WithV2.0.0, you simply set the.guidattribute (on the object itself) toNone.# V1.3.0tml=YAMLTML.get_tml_object(worksheet_tml_str)tml.remove_guid()# V2.0.0tml=Worksheet.loads(worksheet_tml_str)tml.guid=NoneInV1.3.0, each TML object had their own methods for finding and replacing GUIDs. These took the form of.remap_<object_type>_to_new_fqn()and.change_<object_type>_by_fqn(), replacing<object_type>for the underlying data source which maps into the object you're operating on. These methods modify the underlying object.InV2.0.0, we supply a single method to help add thefqnkey to your TML document when referencing source tables or connections that share a name.See disambiguationfor additional information.For example, the below example shows adding the Table FQN references in a Worksheet.# V1.3.0name_guid_map={"Table 1":"0f814ce1-dba1-496a-b3de-38c4b9a288ed","Table 2":"2e7a0676-2acf-4700-965c-efebf8c0b594"}tml=YAMLTML.get_tml_object(worksheet_tml_str)tml.remap_table_to_new_fqn(name_to_fqn_map=name_guid_map)# - or -tml.change_table_by_fqn(original_table_name="Table 1",new_table_guid="0f814ce1-dba1-496a-b3de-38c4b9a288ed")# V2.0.0fromthoughtspot_tml.utilsimportdisambiguatetml=Worksheet.loads(worksheet_tml_str)tml=disambiguate(tml,guid_mapping=name_guid_map)Notes onThoughtSpot Modeling LanguageTML is implemented in the YAML 1.1 spec.When importing a TML file, if theguidmatches to an existing object, then that object will be updated. If theguidis missing or does not match an object, a new object is created with a new GUID.Want to contribute?We welcome all help! :heart: For guidance on setting up a development environment, see ourContributing Guide.