Package: iidfile
IID fileformat (.iid).iidfiles contains a list of masks and labels associated with an image. It is a file format for storing image segmentations mapped toIndividual IDentifiers(IIDs).This package provides methods for loading, saving, searching and computing data from.iidfiles.Why? Because it was needed to map thisae73edb74571e4e2.InstallationpipinstalliidfileQuick ExamplefromiidfileimportIIDFilemask=np.ones(shape,dtype=np.bool)bbox=(minr,minc,maxr,maxc)# Create iidfileiidfile=IIDFile()iidfile.add(address=b'foo',domain=b'bar',bbox=bbox,mask=mask)iidfile.save('path/file.iid')# Load iidfileiidfile=IIDFile(fpath="path/file.iid")entries=iidfile.fetch(everything=True)OverviewIID files are not image files, rather they store labeled segments that maps a image file. The IID files have no set resolution, instead every segments has a bounding box that places the segment mask on the image it maps. There is no limit to the number segments or the ways that they can overlap. Overlaps has no depth hierachy, there is no concept of in front or behind.EntriesInternally an.iidfile contain a lookup table, where eachentrystores an IID and a segment pair. Each entry have a key, which correspond to their position in the lookup table.# Add entry (iid, segment) to iidfileiidfile.add(address=b'foo',domain=b'bar',bbox=bbox,mask=mask)# Get the first entryentry=iidfile.fetch(everything=True)[0]# Attributeskey=entry.keyaddress=entry.iid.addressdomain=entry.iid.domainmask=entry.seg.mask()bbox=entry.seg.bboxSegmentsSegments are binary masks placed by a bounding box. These masks are paired to a single IID each, and thereby maps the part of the image they cover to that spesific IID.IID (Individual IDentifiers)IIDs are labels composed of anaddressand adomain. Both the domain and address is stored as raw byte strings. There is no length limit to this bytestring. The domain defines the context (what language/protocol) and address defines the name of the label in that context.The purpose of IIDs is to provide globally unique labels that can link segments across.iidfiles.# Masks are binary numpy buffersmask=np.zeros(shape,dtype=np.bool)# Bounding boxes determine the placement of the mask in the image coordinate spaceh,w=mask.shapebbox=y,x,y+h,x+w# minr, minc, maxr, maxciidfile.add(address=b'foo',domain=b'bar',bbox=bbox,mask=mask)FeaturesAn IID-file is structured with a lookup table that maps the memory location of the entry data. This enables selective loading of file-content. Selective loading can limit the parsing needed when searching through large.iidfiles. This python implementation usesmmapto parse spesific parts of the file-buffer.GroupsGroups are sets of entries, they enable faster and targeted loading of segments.# Adding entry to group on creationiidfile.add(address=b'foo',domain=b'bar',bbox=bbox,mask=mask,group='sys')iidfile.save('path/file.iid')# Only load segments in specified groupiidfile=IIDFile(fpath='path/file.iid')entries=iidfile.fetch(groups=['sys'])MetadataAn embedded dict stores any additional meta data, this can be camera parameters or info about the external image. The dict must be JSON serializable.metadata=iidfile.meta.dataSegments and regionsInternally segments are split up into smaller regions of connected patches, this compresses the segment mask buffer, especially when the mask is large and scattered.ReferencesUse docstring for details.IIDFileadd- Add an entry (iid and segment) to filesave- Save file to diskfetch- Selective loading of entrieslook_for- Search for iidsfilter- Filter loaded entriesat- Get segments intersecting at positionregion- Get segments intersecting with a given bounding boxcompute_overlap- Computes an overlap graph from loaded segmentsExamples📦Download some example .iid filesAssign segments and save to diskfromiidfileimportIIDFileiidfile=IIDFile()# Creates empty iidfiledata=[(b'tree',numpy_buffer_1,xy_1),(b'leaf',numpy_buffer_2,xy_2),(b'root',numpy_buffer_3,xy_3)]foriid,mask,xyindata:# Format bbox to minr, minc, maxr, maxcx,y,w,h=*xy,mask.shape[1],mask.shape[0]bbox=(y,x,y+h,x+w)iidfile.add(address=iid,domain=b'example',bbox=bbox,mask=mask)iidfile.save(fpath='example.iid')Fetch content from saved file# Load everythingiidfile=IIDFile(fpath='example.iid')entries=iidfile.fetch(everything=True)# Only load entries in group 'sys', and only load their iids,# this will not load their segments.iidfile=IIDFile(fpath='example.iid')entries=iidfile.fetch(groups=['sys'],iids=True)# Load all entries and only load their segments,# this will not load their iids.iidfile=IIDFile(fpath='example.iid')entries=iidfile.fetch(all_keys=True,segs=True)# Only load the first 50 entries.iidfile=IIDFile(fpath='example.iid')entries=iidfile.fetch(keys=range(50),iids=True)Query, filter and look for content# Finds all segments intersecting at x:400, y:600.iidfile=IIDFile(fpath='example.iid')entries=iidfile.at(400,600)# Only fetch entries in the group 'sys', do not load thier# iids and find which intersects with the bounding box.iidfile=IIDFile(fpath='example.iid')iidfile.fetch(group=['sys'],iids=False)entries=iidfile.region(bbox,only_loaded=True)# Check if iid addressess exists in file.iidfile=IIDFile(fpath='example.iid')entries=iidfile.look_for(addresses=[b'tree',b'leaf'])# Filter entries in file, this return entries in the group# 'rep' with a segment area of 4000 and greater.iidfile=IIDFile(fpath='example.iid')iidfile.fetch(everything=True)entries=iidfile.filter(groups=['rep'],area=(4000,None))FileformatDescribed with psuedo-BNF. Types contained in brackets{ }represents lists.Common typeschar      ::= 1 byte
uint8     ::= 1 byte
uint16    ::= 2 byte                       short
uint32    ::= 4 byte                       32-bit unsigned integer
len       ::= uint32                       length in bytes
string    ::= len { char }
json      ::= string                       raw json string
bufloc    ::= uint32 uint32                buffer location, offset and lengthFile structurefile      ::= header
              lut                          lookup table (LUT) from key to iid and segment
              iids                         
              meta                         meta data as json
              groups                       groups
              segments                     segments

header    ::= version                      format version number
              rformat                      TODO: consider removing?
              bufloc_lut                   location of lookuptable
              bufloc_iids                  location of iid data (relative)
              bufloc_meta                  location of meta data
              bufloc_groups                location of groups
              bufloc_segs                  location of segments

version   ::= uint32
rformat   ::= uint32

lut       ::= { key iid seg }               
iids      ::= { key len len bytes bytes }  key, len, len, domain, iid
meta      ::= json                         metadata and other info

groups    ::= len json { group }           len, group list, group data
group     ::= { key }                      list of LUT keys

key       ::= uint32                       LUT key, maps to iid and segment
iid       ::= bufloc                       location of iid
seg       ::= bufloc                       location of segment

segments  ::= { key bbox area { region } }
region    ::= len bbox { byte }            len, bbox, mask buffer

area      ::= uint32                       counted in pixels
bbox      ::= uint16*4                     minr, minc, maxr, maxc