It is not unreasonable to think that such interests existed prehistorically, just difficult to know. Today, many astronomers are fully supported to study phenomena occurring at the farthest reaches of the universe. While such extreme specialization is possible in complex societies, I think there is evidence for partial specialization at lesser levels of organization. In Peru, Guaman Poma, and the anonymous author of the Huarochiri manuscript describe village level sky watchers. It is not clear how specialized these individuals were, but they certainly occupied a special niche in villages with rather limited resources.
Acquisition of knowledge requires time and resource expenditure.
The more time that an individual can dedicate to study, the more rapidly the knowledge base expands. A general expectation of the annual motion of the sun might be acquired rather casually, and known to most farmers. Precision and prediction requires an individual to spend time and effort making systematic observations, and organizing the resulting data. As a result, the more resources given to support such sky watchers, the more time they have to develop it as a specialized activity. In addition to being time-keepers, these village sky watchers were accepted as sources for other important information.