You can get the InventoryHolder of an inventory to get the Block or Entity that holds an inventory.  Is there a reason to explicitly keep a reference to these?