data extraction and some refactoring