more work to catch edge cases for the invisibility field and made search backwards compatible with old data