Package: pyjmap
pyjmappyjmapis a high-level implementation of Nintendo's homemade BCSV/JMap data format. This includes methods to construct, analyze, manipulate, deserialize and serialize proper JMap data. Conversion between CSV and BCSV files is also supported. The reverse-engineered specifications of the file format can be accessed on theLuma's Workshop wiki. This flatbuffer-like data type was used in first-party GameCube and Wii games. As the field/column names are hashed, a lookup table needs to be used to retrieve proper field names. For this, the library provides hashtable implementations forSuper Mario Galaxy,Super Mario Galaxy 2,Luigi's MansionandDonkey Kong Jungle Beat.SetupThis library requiresPython 3.6 or newer. You can use pip to installpyjmap:pipinstallpyjmapCommand usageCommand line operations to convert between JMap and CSV files are supported. The CSV files are required to be in a special format that has been found in some leftover source files fromSuper Mario Galaxy 2. That format is described down below.You can dump the contents of a BCSV/JMap file to a CSV file using:pyjmaptocsv[-le][-jmapencJMAP_ENCODING][-csvencCSV_ENCODING]{smg,dkjb,lm}JMAP_FILE_PATHCSV_FILE_PATHProper CSV files can be converted back to BCSV/JMap files using:pyjmaptojmap[-le][-jmapencJMAP_ENCODING][-csvencCSV_ENCODING]{smg,dkjb,lm}CSV_FILE_PATHJMAP_FILE_PATHIfleis set, the data is expected to be stored using little-endian byte order.jmapencspecifies the encoding of strings in the JMap data and it defaults toshift_jisx0213.csvencis the encoding of the CSV file and it usesutf-8by default. The hash lookup table is specified byHASHTABLE. Supported values aresmgforSuper Mario Galaxy,lmforLuigi's Mansion,smsforSuper Mario SunshineanddkjbforDonkey Kong Jungle Beat.Library usageThe library provides various high-level operations to deal with JMap data. Below is some example code showing the fundamentals ofpyjmap. Look atjmap.pyfor more information about the different methods.importpyjmap# A hash lookup table is required to retrieve the proper names for hashed fields:hashtbl_smg=pyjmap.SuperMarioGalaxyHashTable()# Lookup table for Super Mario Galaxy 1/2hashtbl_sms=pyjmap.SuperMarioSunshineHashTable()# Lookup table for Super Mario Sunshinehashtbl_lm=pyjmap.LuigisMansionHashTable()# Lookup table for Luigi's Mansionhashtbl_dkjb=pyjmap.JungleBeatHashTable()# Lookup table for Donkey Kong Jungle Beat# Create JMapInfo data from files and print number of entriesinfo=pyjmap.from_file(hashtbl_smg,"GalaxySortIndexTable.bcsv",big_endian=True)# Big-endian is True by defaultinfo_from_csv=pyjmap.from_csv(hashtbl_smg,"GalaxySortIndexTable.csv")# Load data from CSV fileprint("Number of entries:%d"%len(info))# >> Number of entries: 55# Print fieldsforfieldininfo.fields:print(field)# >> name# >> MapPaneName# >> OpenCondition0# >> OpenCondition1# >> OpenCondition2# >> PowerStarNum# >> GrandGalaxyNo# Checking if a field existsprint("MapPaneName"ininfo)# >> Trueprint("StageName"ininfo)# >> False# Getting information about a fieldfield=info.get_field("MapPaneName")# Get field by namefield=info.get_field(0x7991F36F)# Get field by hashprint("[%08X]"%field.hash)# >> [7991F36F]print(field.name)# >> MapPaneNameprint(field.type)# >> JMapFieldType.STRING_OFFSETprint("0x%08X"%field.mask)# >> 0xFFFFFFFFprint(field.shift)# >> 0# Manually-specified offsets and bit-packed datacollision_pa=pyjmap.JMapInfo(hashtbl_smg)collision_pa.manual_offsets=Truecollision_pa.create_field("camera_id",pyjmap.JMapFieldType.LONG,0,mask=0x000000FF,shift_amount=0,offset=0)collision_pa.create_field("Sound_code",pyjmap.JMapFieldType.LONG,0,mask=0x00007F00,shift_amount=8,offset=0)collision_pa.create_field("Floor_code",pyjmap.JMapFieldType.LONG,0,mask=0x001F8000,shift_amount=15,offset=0)collision_pa.create_field("Wall_code",pyjmap.JMapFieldType.LONG,0,mask=0x01E00000,shift_amount=21,offset=0)collision_pa.create_field("Camera_through",pyjmap.JMapFieldType.LONG,0,mask=0x02000000,shift_amount=25,offset=0)# Creating an exact copy of the datacopied=info.copy()# The following creates a new field called CometMedalNum which uses the LONG data type. The field's default value# that is applied to all fields is -1. The optional bitmask and shift amount are 0xFFFFFFFF and 0, respectively.copied.create_field("CometMedalNum",pyjmap.JMapFieldType.LONG,-1,mask=0xFFFFFFFF,shift_amount=0)# This removes the field OpenCondition2 and its data in all entries.copied.drop_field("OpenCondition2")# Accessing entries directlyfirst=copied[0]# Get first entry from copied datalast=copied[-1]# Get last entry from copied data# Adding and deleting entriesnew_entry=copied.create_entry()# Creates a new entry with default data for all fieldsdelcopied[-3:]# Delete the last three entries from the copied data# Iterate over all entries and set GrandGalaxyNo to 0forentryincopied:entry["GrandGalaxyNo"]=0# Sort entries by name in lexicographic descending orderinfo.sort_entries(lambdae:e["name"].lower(),reverse=True)# Get all entries whose name start with "Koopa"forentryinfilter(lambdae:e["name"].startswith("Koopa"),info):print(entry)# >> {'name': 'KoopaJrShipLv1Galaxy', ... }# >> {'name': 'KoopaBattleVs3Galaxy', ... }# >> {'name': 'KoopaBattleVs2Galaxy', ... }# >> {'name': 'KoopaBattleVs1Galaxy', ... }# Write data to filespyjmap.write_file(info,"GalaxySortIndexTable_edited.bcsv",big_endian=True)# Pack and write binarypyjmap.dump_csv(copied,"GalaxySortIndexTable_copied.csv",encoding="utf-8")# Dump CSV content# Pack as little-endian bufferpacked_copied=pyjmap.pack_buffer(copied,big_endian=False)Data typesThe following field data types are supported:IdentifierCSV typeDescriptionJMapFieldType.LONGInt32-bit integerJMapFieldType.UNSIGNED_LONGUnsignedInt32-bit unsigned integerJMapFieldType.SHORTShort16-bit integerJMapFieldType.CHARChar8-bit integerJMapFieldType.FLOATFloatsingle-precission floatJMapFieldType.STRINGEmbeddedStringembedded SJIS string (occupies 31 bytes at max)JMapFieldType.STRING_OFFSETStringSJIS string (not supported in Luigi's Mansion)CSV formatThe CSV format is based on the format of known source files that were left in the files ofSuper Mario Galaxy 2:All CSV files are comma-delimited and use quote-marks for quoted cell strings. Quoting is only used when necessary.The first CSV-row contains the field descriptors. A field descriptor always consists of three components that are separated by double-colons: the field's name, the data type and the default value. All existing CSV types are described in the previous section and are case sensitive!The default value for strings is 0 and is always ignored. It is only kept for syntax.If an entry's field data is empty, the default value will be used.The field name may be a hash if it's a hex-string encapsulated between two square brackets (for example[DEADBEEF])Here is an example of a properly-formated CSV file:name:String:0,MapPaneName:String:0,OpenCondition0:String:0,OpenCondition1:String:0,OpenCondition2:String:0,PowerStarNum:Char:0,GrandGalaxyNo:Char:0
AstroGalaxy,dummy,,,,0,0
AstroDome,dummy,,,,0,0
LibraryRoom,dummy,,,,0,0
PeachCastleGardenGalaxy,dummy,,,,0,0
EpilogueDemoStage,dummy,,,,0,0