Managing social debt in large software projects

While software engineering has traditionally focused on artifacts---code, processes, tools and so forth---there has been an increasing emphasis in recent years on the people and teams that build those artifacts. This can be traced back to the origins of software engineering as a discipline. For example, in 1968 Melvin Conway described, in a short paper, what has now become known as "Conway's Law". This "law" states that the structure of a software system reflects the social structure of the organization that produced it. Conway cited several pieces of anecdotal evidence to support his claim; for example a COBOL compiler that was developed by 5 people and ran in 5 phases, and an ALGOL compiler that was developed by 3 people and ran in 3 phases. This notion-that the social structure of an organization matters---has been shown over and over in the 50+ years since Conway first wrote this article. Since there are countless ways to structure organizations and countless ways to structure systems, it stands to reason that some of these structures are sub-optimal. Furthermore, it stands to reason that some mappings of organizational designs onto system designs will also be sub-optimal. Those sub-optimal mappings are instances of what we have come to call social debt. We can think of social debt as a kind of socio-technical incongruence. Ideally you want socio-technical congruence in your project: components that are tightly connected should be developed by teams that are in constant communication with each other, ideally co-located. Components that have little or no interaction can be developed by teams that are similarly isolated from each other. In both of these cases we have congruence between the social structures and the technical structures. While congruence does not guarantee the absence of social debt, it certainly helps. Furthermore, in the world of distributed development, misalignments between a project's social or organizational structure and its architecture might be much harder to detect and more insidious. But debt can appear for many reasons, and take many forms. This, then suggests several interesting research questions: how do we identify social debt, what are its consequences, and how do we manage it? Architects need to be aware of smelly software development communities and of the potential misalignments between communities and software structures. They need this awareness so that they can detect and correct those misalignments before they cause too much damage. This is a relatively new duty for architects. We have always known that architects need to be much more than just a "technical lead" in a project but now it seems that they must also be an active community shepherd as well. We have chosen the term "shepherd" consciously, because a shepherd guides and cares for a flock. In this talk I will discuss how you can identify "community smells"---sub-optimal community structures---and how you can monitor these smells just as you would monitor technical debt, test coverage, feature velocity or any other aspect of the development process. Finally, I will discuss how you, as an architect, can take proactive steps to avoid or repair these smells, before they doom your project to failure.