refactoring, debugging the single problem left, after this scaling up to complete database