adding rest of code crawler and twitter bot