# Combining Pragmatism and Critical Realism in ICT4D Research: An e-Resilience Case Example

Richard Heeks, Angelica V. Ospina, P. J. Wall

► To cite this version:

Richard Heeks, Angelica V. Ospina, P. J. Wall. Combining Pragmatism and Critical Realism in ICT4D Research: An e-Resilience Case Example. 15th International Conference on Social Implications of Computers in Developing Countries (ICT4D), May 2019, Dar es Salaam, Tanzania. pp.14-25, 10.1007/978-3-030-19115-3_2. hal-02281318

HAL Id: hal-02281318

https://inria.hal.science/hal-02281318

Submitted on 9 Sep 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Combining Pragmatism and Critical Realism in ICT4D Research: An e-Resilience Case Example

Richard Heeks<fnref n="1" />[0000-0002-4551-2208], Angelica V Ospina<fnref n="1" /> and P.J. Wall<fnref n="2" />[0000-0002-5859-4425]

**Abstract.** ICT4D research is strongly oriented to practice but hardly ever explicitly uses the research paradigm of pragmatism. We argue that, though highly-relevant to ICT4D, pragmatism suffers some shortcomings in terms of its philosophy of the world, explanatory power, truth-testing, and values. We suggest that “pragmatist-critical realism” – a novel research paradigm combining pragmatism and critical realism – can address these shortcomings and provide a valuable foundation for ICT4D research; particularly action-oriented research. We outline a four-step operational methodology for pragmatist-critical realism based on a research project that created an “e-resilience” action plan applying ICTs to strengthen resilience of farming communities in Uganda. We hope other action- and design-oriented ICT4D researchers will be encouraged to assess whether pragmatist-critical realism could form a useful basis for their future research.

**Keywords:** ICT4D, pragmatism, critical realism, methodology, resilience

## 1 Introduction

The application of information and communication technologies to development (ICT4D) is highly practical. It involves real-world actions in the design, construction, implementation and use of digital technologies for the furtherance of development goals. ICT4D research is thus heavily engaged with practice since it must generally study, and sometimes directly involve these real-world actions.

While ICT4D research is thus often pragmatic – meaning oriented to practice – it is rarely pragmatist – meaning guided by the philosophy of pragmatism. Lack of engagement with research philosophies is a general characteristic of ICT4D research [16]. However, there are particular challenges with pragmatism: its dissimilarity to other paradigms, the diversity of views it contains, and its emergent shortcomings when applied. Our aim in this paper is: to explain one view of pragmatism; to argue that it may fruitfully be combined with the philosophy of critical realism in an attempt to address its shortcomings; and to describe – as a set of methodological steps – an initial example of applying this “pragmatist-critical realism” in ICT4D research.

In the section that follows, we review key features of pragmatism and its combination with critical realism. We then explain the focal research case: an action-oriented research project seeking priorities for application of ICTs to improve

<footnote n="1">¹Centre for Development Informatics, University of Manchester, UK
richard.heeks@manchester.ac.uk, angelica.v.ospina@gmail.com</footnote>

<footnote n="2">²ADAPT Centre, Trinity College Dublin, Dublin, Ireland wallp2@tcd.ie</footnote>
resilience of a farming community in Uganda. The application of pragmatist-critical
realism in the research is described before conclusions are drawn.

## 2 Pragmatism, Critical Realism and Pragmatist-Critical Realism

Research paradigms can often be understood as a kind of continuum. At one end is positivism: "The key idea of positivism is that the social world exists externally, and that its properties can be measured through objective methods rather than being inferred subjectively through sensation, reflection or intuition." [8:p51]. At the other end is interpretivism: "the view that 'reality' is not objective and exterior, but is socially constructed and given meaning by people in their daily interactions with others ... [it] focuses on the ways that people make sense of the world especially through sharing their experiences with others via the medium of language." [ibid.:p52]. In between are a number of middle-ground variants including critical realism: "the philosophical stance that what we experiences are some of the manifestations of the things in the real world, rather than the real things" [30:p714] which combines positivism's view that there is an objective reality with interpretivism's view that our knowledge of that reality is socially-constructed.

A first challenge in explaining pragmatism is that it does not readily sit on this continuum because its fundamental interest is orthogonal to the continuum. The other research paradigms differentiate themselves around metaphysical questions of ontology: what the paradigm understands to be the nature of reality; and epistemology: what the paradigm understands about how we construct and evaluate knowledge about that reality. But pragmatism remains rather uninterested in such issues, "it is ... aimed at producing useful knowledge rather than understanding the true nature of the world" [21:p297] and "argues that concepts are only relevant where they support action. It considers research starts with a problem, and aims to contribute practical solutions that inform future practice" [30:p724].

A second challenge is the many varieties of pragmatism that exist, with variation along multiple dimensions including the shortcomings discussed below. To take just one example, when analysts have sought to extract a metaphysical position from the writings of pragmatism's key thinkers, Dewey has been argued to be a positivist, Peirce a realist, and Rorty an anti-realist relativist [1, 25].

A third challenge is lack of engagement of ICT4D research with pragmatism. A search for literature identifying itself as ICT4D research and mentioning pragmatism found relatively few items overall, mostly using the term in its lay sense: asserting the practical nature of ICT4D but not linking this to the philosophy. Of the remaining 11 items<fnref n="1" />, seven make just a brief assertion that they are guided by the philosophy of pragmatism but without further explanation or exploration; and four provide only a little more detail identifying pragmatism as a relevant frame for ICT4D action and/or design-science-based research seeking to construct knowledge of practical utility.

<footnote n="1">¹ Notably eight of the eleven items linking ICT4D research to the philosophy of pragmatism were from South Africa-based authors.</footnote>
This very limited depth and breadth – without direct reference to the works of pragmatist philosophers – provides little basis for an understanding of what pragmatism means in ICT4D.

Within the limits imposed by these challenges, we identify three criticisms of pragmatism [3, 10, 20].

First, that its ontological and epistemological agnosticism means it can place anywhere on the continuum described above – as just noted, pragmatist “research may have considerable variation in terms of how ‘objectivist’ or ‘subjectivist’ it turns out to be” [30:p143] – or it can place nowhere; simply not engaging with ontology or epistemology. But having no set or defined position on ontology and epistemology is inconsonant with the actualities of research: as researchers, we all have a position on this, even if implicit, that guides our analysis. And that position is relatively invariant: we cannot jump back and forth between being a positivist and an interpretivist just because that proves differentially useful for different research; any more than one can flip-flop between being a Christian and a Muslim. So it will be appropriate for any research paradigm to include a metaphysical position or at least require researchers to clarify their beliefs about “the true nature of the world”.

Second, while pragmatism aspires to “transferability” of knowledge from one context to another [23], it provides no necessary explanatory foundation for this. It can demonstrate that a particular solution has worked in a particular context, but it cannot explain why the solution has worked; e.g., thus severely limiting the confidence with which solutions can be recommended for other contexts. Put another way, pragmatism’s belief that the truth of any knowledge lies in its practical utility provides a very shallow basis for that truth: a house built on sand that is easily washed away; particularly as utility can only be proven or disproven post-hoc by implementation.

Third, pragmatism is relativist in its judgement of practice, despite the strong engagement of many of its founding fathers with questions of ethics [ibid.]. It orients itself towards “what works” but has no inherent values or guidance about the implications of this. Issues of whose problems are attended to, of who wins and who loses from practice, may be addressed but pragmatism provides no inherent judgement on the answers.

## 2.1 Pragmatist-Critical Realism

In this paper, we argue not merely that critical realism is commensurable with pragmatism but that explicitly recognising their intersection as a variant research paradigm – pragmatist-critical realism (PCR) – will help address the shortcomings of pragmatism alone. PCR would be defined as a research paradigm based on our socially-constructed experience of the manifestations of an external, independent reality that aims to provide practical and emancipatory solutions to problems of inequality.

Generally, we argue commensurability of critical realism and pragmatism on three grounds. First, philosophically that pragmatism’s orthogonality to the metaphysical paradigm continuum – e.g. its agnosticism about ontological issues – means there is
no barrier to combining it with a paradigm on that continuum. Second, teleologically, that pragmatism and critical realism can be seen to share a similar purpose as a “third way” between positivism and interpretivism: critical realism in an ontological and epistemological sense [31]; pragmatism in a methodological sense [23]<fnref n="2" />. Third, developmentally that 21st century writings on pragmatism – including attempts to respond to the three shortcomings through some reinterpretation of pragmatism – are actually fusing the ideas of critical realism and pragmatism; up to the point where a very few writers have now made explicit this potential fusion [6, 19].

Specifically, in terms of the three shortcomings identified above, PCR would assert a stable ontological and epistemological position: that of critical realism<fnref n="3" />. This adopts a three-level stratified ontology (see Figure 1). The real domain has an objectivity independent of human thought and includes generative mechanisms: “causal structures that generate observable events” [17:p911]. The actual domain contains events: “specific happenings resulting from causal mechanisms being enacted in some social and physical structure within a particular ... context” [33:p939]. The empirical domain encompasses human experiences and observations of the events generated within the actual, with those experiences being not objective but shaped by their context.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Stratified Ontology of Critical Realism [22]</figcaption></figure>

PCR would look to the causal mechanisms of the real domain to provide a stronger foundation for explanation of the practical outcomes observed from any intervention. The posited existence of such mechanisms would more-readily allow a convincing basis for cross-context generalisation of those outcomes, and a greater credibility for building a consensus around the value of particular interventions (consensus-building

<footnote n="2">² Some authors go further and interpret this as a metaphysical commensurability e.g. “Dewey [7] describes pragmatism to be based on both realist and idealist metaphysics. Pragmatism accepts things and events as existing independent of any observers, but at the same time emphasizes reason and thought as originators of elements in the external world. Goles & Hirschheim [13] describe pragmatism as taking a middle or dual position between positivist and interpretivist ontologies.” [12:p141].</footnote>

<footnote n="3">³ Material here on critical realism summarises from Heeks & Wall [16], which should be referred to for further detail.</footnote>
being a key test of validity within pragmatism<fnref n="4" />). Phrased differently, PCR integrates two bases for truth-testing particular knowledge-claims – which could be a concept, idea or intervention (see Figure 2). From critical realism, validity is strengthened by a gradual accretion of triangulated evidence in different contexts that supports the existence of proposed mechanisms. From pragmatism, validity is strengthened by gradual accretion of practical applications of knowledge in different contexts that achieve the intended practical purposes. Together – both sides of Figure 2 being active during action research – validity is strengthened by gradual accretion of triangulated evidence in different contexts that proposed mechanisms achieve intended practical purposes.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Knowledge-Building under PCR (adapted from Johnson & Duberley [19])</figcaption></figure>

In terms of the third shortcoming, PCR moves pragmatism away from its relativistic view of purposes by drawing on the axiology of critical realism: what it does and does not value in research. Critical realism is “specifically driven by the values of emancipation. This means recognising the way in which the social structures and mechanisms of the real domain can sometimes serve to generate events and processes that are oppressive and outcomes that are unequal” [16:p4]. PCR thus delimits it purposes and judges practice on the extent to which it generates events and processes that counter oppression and on the extent to which it generates pro-equity outcomes.

Finally, we can flip the main line of argument, and look for ways in which pragmatism addresses shortcomings within critical realism or, at least, adds value to critical realism. More work is needed here but one dimension to this is critical realism’s link to practice. While the philosophy is sympathetic to and engaged with practice [28], it is not always applied in a practical way – for example, being applied to retrospective analysis of projects [24]; and sometimes applied to theory-oriented research [32]. The integration of pragmatism strengthens this engagement, and strongly encourages that ICT4D research be action research. This practical action

<footnote n="4">⁴ We also believe that a realism-based approach to ICT4D work is more likely to build con- sensus within the ICT4D community than a constructionist approach given the great bulk of publication in the field adheres to some form of realism [14], suggesting a preponderance of realist worldviews within the ICT4D research community.</footnote>
would also be the only way to deliver on critical realism's axiological aspirations to
emancipate those afflicted by structures of oppression and inequality.

3 Pragmatist-Critical Realist Methodology Overview

There are a number of potential starting points to provide the structure for a pragmatist-critical realist methodology. These are relatively easy to identify for critical realist methodology with the recent appearance of operational guides including:

* Four-step methodology: Description of events, Identification of entities and associations, Abduction (theoretical re-description), Retroduction [4].

* Six-step methodology: as four-step plus Analysis of mechanisms (affordances), and Assessment of explanatory power of mechanisms [5].

"Pragmatist methodology" as such is rarely identified but it is associated with three threads:

* Mixed-methods methodology [9], combining qualitative and quantitative methods in some way<fnref n="5" />.

* Action research methodology [29], such as the action research cycle Plan – Act – Observe – Reflect [34].

* Design science research methodology [18], such as the three-cycle approach of Relevance – Design – Rigour [ibid.] or the six-step approach of Problem – Solution – Design/Development – Demonstration – Evaluation – Communication [27].

In this case, we have chosen to use the action research methodology as the skeleton into which to integrate elements from the other methodologies but we acknowledge this as just one possible approach that could be taken.

We operationalise this in Section 4 through a case study relating to resilience-building; a topic that has risen sharply up the international development agenda in recent years; particularly in terms of strengthening the resilience of communities [35]. We define resilience as “the ability of a system to withstand, recover from and adapt to short-term shocks and longer-term change” [15:p75], and undertook action-oriented research on resilience-building in Uganda and Costa Rica with a particular focus on community resilience to climate change.

The project reported here was based in the Uganda coffee-farming region of Mbale; around Mount Elgon. This has been particularly affected by climate change manifestations: more frequent droughts, a rise in average temperature, and increase in heavy rainfall and consequent floods and landslides [26]. Coffee farming is especially susceptible to climate change and so increasing resilience of farming communities to this and other wider shocks and change is seen as a development priority [ibid.].

We worked with farmer cooperatives in four areas around Mount Elgon: Bukalasi, Bumayoga, Busamaga and Konokoyi; those cooperatives belonging to the Gumutindo

<footnote n="5">⁵ Though arguably the association of mixed methods ought to be stronger to critical realism – where they can be seen as a requirement for validation of mechanisms [16] – whereas under pragmatism, methods are judged on their practical value rather than there being some inherent requirement to mix quantitative and qualitative [2].</footnote>
Coffee Cooperative Enterprise (GCCE). GCCE was set up in 1998 to try to improve the incomes and wider livelihoods of smallholder farmers. At the time of our project, it had 7,000 farmer-members and was being supported by Lutheran World Relief (LWR). LWR and GCCE both had strategic concerns about climate change and resilience. But they also saw opportunities based on the growing use of ICTs, mobile phones particularly, in the Mbale region. It was for this reason that we were engaged with the project described next, with what we identified as a pragmatist-critical realism-based approach.

# 4 Application of a Pragmatist-Critical Realist Methodology

Below, we explain the steps of an applied PCR methodology both in general and then case study-specific terms.

## 1. Plan

### 1a. Clarify Generic Problem and Emancipatory Purpose

In this step, researchers specify some emancipatory purpose for the action research project relating to a broad problem – e.g. lack of freedoms, equality or justice for a group – including details of for whom and in relation to what the problem exists. For the Uganda project, the generic problem was lack of resilience of low-income farming communities, hampering their ability to cope with environmental and other changes. The overall purpose was thus to increase community resilience. As with later steps, this could often be a participative, even bottom-up activity. In Uganda it was determined as part of a pre-existing and wider project of work that had engaged the community.

### 1b. Identify Specific Problem and Purpose of Intervention

In this step, the researchers identify a specific problem currently seen as hampering the overall emancipatory purpose. In the Uganda case, this was the increasing impact of climate change and variability on farmer livelihoods and, alongside this, the growing role of ICTs. One could either interpret current limited use of ICTs to support resilience as a problem, or flip this to see potential greater use in future as an opportunity. The specific purpose of the intervention then became to create an “e-resilience action plan”: a set of priorities that would make best use of ICTs in order to increase community resilience, especially resilience to climate change.

### 1c. Identify Proposed Mechanisms for Solution

PCR favours an abductive approach: iterative combination of inductive and deductive reasoning moving back and forth between data and theory<fnref n="6" />. The starting point for iteration could be data (an inductive-first approach in which mechanisms only emerge out of action research) or theory (a deductive-first approach in which mechanisms

<footnote n="6">⁶ The emphasis on iteration is slightly different from the interpretation of abduction by some critical realists [e.g. 4, 11] which emphasises induction more than iteration, seeing abduction as an activity after fieldwork that moves from the empirical data to the re-description of that data “using theoretical concepts” [ibid.:p188].</footnote>
would precede and guide action research)<fnref n="7" />. For the Uganda project, we chose the latter, conceptualising resilience as a set of eight attributes known as the RABIT (Resilience Assessment Benchmarking and Impact Toolkit) framework<fnref n="8" />. Three foundational: robustness, self-organisation, learning; and five enabling: redundancy, rapidity, scale, diversity and flexibility, and equality. Attributes are understood as synonymous with mechanisms: “potentials of a system which, if enacted, would lead to events that would in turn impact the system” [15:p78].

Our primary purpose of PCR-based research would not be to verify the presence of the proposed mechanisms (that would be a secondary purpose) but to understand their nature in the specific context in order to guide practical action. With that in mind, we operationalised each of the attribute-mechanisms as a set of three illustrative markers: “observable characteristics for each attribute again derived from the ecological literature on resilience” [ibid.:p78].

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Example visualisation of findings [26]</figcaption></figure>

**1d. Design Intervention and Methods**

This step involves the design of the mechanisms-based action intervention, including any associated research methods for implementation and evaluation. Consistent with the tenets of both critical realism and pragmatism, PCR would favour mixed methods action research. In the Uganda case, and in order to build the e-resilience action plan, we designed a two-part process:

* First, benchmarking the current state of community resilience generally (i.e. the strength or weakness of resilience attributes) and then e-resilience specifically (i.e. the current level and nature of impact of ICTs on the attributes of community resilience). Data for this part was to be gathered through: i) a questionnaire

<footnote n="7">⁷ In the terminology of critical realism, the former is a retroduction-first approach, the latter a retrodiction-first approach [36].</footnote>

<footnote n="8">⁸ Material here on resilience summarises from Heeks & Ospina [15], which should be referred to for further detail.</footnote>
survey undertaken with a cross-sectional sample of farmers; ii) semi-structured interviews with a purposive sample of farmers, community knowledge workers and cooperative staff; iii) focus groups with farmers and with cooperative staff.

* Second, analysing, visualising and reporting back these findings to the community to enable the action plan to be developed. This was to be undertaken through two participative workshops: one in Kampala intended to identify national-level actions; one in Mbale intended to identify local-level actions.

## 2. Act

In this stage the intervention is enacted. In the Uganda case, we surveyed 54 farmers, undertook 16 interviews and five focus groups (three with farmers, two with cooperative staff). Findings were then visualised (see Figure 3), and presented to the two participative workshops.

## 3. Observe

In this stage, the outcome of the action research intervention is observed and evaluated – both its research and its action components – with the evaluation of the latter potentially requiring additional research. In the Uganda case, alongside many interim outcomes such as the visualisations of findings, the final outcome was the e-resilience action plan. A sample of this is shown in Table 1.

<table><caption>**Table 1.** Part of e-Resilience Action Plan</caption><thead><tr><td>Attribute/Marker</td><td>Intervention</td></tr></thead><tbody><tr><td>Rapidity/Rapid Resource Mobilisation</td><td>Develop effective early warning system combining diverse communication methods and technologies (SMS, radio, etc.)</td></tr><tr><td>Scale/Resource Access and Partnerships</td><td>Foster farmers' ability to make use of external weather information from broader-scale organisations such as FEWSNET</td></tr></tbody></table>

## 4. Reflect

In this stage, the validity of the knowledge-claims is assessed using the two-part approach shown in Figure 2: via iterative cycles of abductive reasoning between data and mechanisms, and also through evaluation of the intervention outcomes against emancipatory purposes. This latter would orient towards measures of freedom and equality, and to the “cui bono” question of who benefits (and who loses) from the intervention.

In the Uganda case, the former was the stronger of the processes. Findings from this and the parallel Costa Rica case suggest that the framework of attributes does “provide insight into aspects of the system (i.e. community) that all in some way relate to withstanding or recovering from or adapting to short-term shocks and longer-term change. ... The findings show that each attribute identified at least some element of life within the community and some use of ICTs that was not found by any other attribute. Since all relate to coping with shocks or trends, this suggests all are necessary to a resilience framework. There is some evidence that they are sufficient:
the more open-ended group interviews did not throw up anything that could not be
related to one of the RABIT framework resilience attributes.” [15:p89]. The validity
of the proposed mechanisms is further strengthened by the demonstrated
transferability of the framework between two different contexts.

In practical terms, our overall emancipatory purpose was to increase the resilience of low-income farming communities beset by climate change. The intervention did not do that because, of course, this would require enactment of the e-resilience action plan; something which post-dates our particular package of work. However, the plan did identify and prioritise a series of practical resilience-strengthening actions. Some of these were incorporated into the strategic plans of both LWR and Gumutindo Cooperative, and implemented.

5 Conclusions

Pragmatism has much to recommend it as a research paradigm for ICT4D research because of its congruity with ICT4D's practical orientation. However, it has some shortcomings. These can be addressed in various ways but our proposal in this paper is that combining pragmatism with critical realism provides a valuable new philosophical foundation for ICT4D research.

Pragmatist-critical realism is something of a chimera<fnref n="9" />, combining a pragmatist orientation, a realist metaphysics, and a critical axiology. No doubt one could readily fall down a rabbit hole of arguments about logical congruities and exact interpretations. But those joys can be postponed for a later day: here we provide an initial outing for PCR; identifying how it might be operationalised by illustrating its use on a project linking ICTs to community resilience-building in Uganda.

Ideally, that project would have had a stronger action component – putting our
action plan into practice rather than just creating the plan – and we acknowledge this
limitation. We also acknowledge that recognition of our PCR approach was emergent
from and retrospective to the project rather than shaping it from the start. Our work
thus provides a first pass at a PCR methodology; something to be refined in future
through *pre hoc* application to other action research projects.

We have linked PCR strongly to action research; seeing action research as a
preferred element of PCR; conversely, seeing PCR as an appropriate foundation for
action research. For those undertaking ICT4D action research, we thus recommend
they consider pragmatist-critical realism as an underpinning paradigm. We do
likewise for those entering the ICT4D field from related directions, e.g. those
embracing a design science approach and those for whom mixed methods research is
their starting point.

We look forward to further applications, revisions and critiques of the pragmatist-
critical realism paradigm.

Acknowledgements

<footnote n="9">⁹ A mythical beast that combined parts of a lion, goat and snake.</footnote>
The field project in Uganda was supported by funding from Lutheran World Relief and the University of Manchester's Strategic Investment Reserve Fund. We also thank two anonymous reviewers of an earlier version of this paper.

References

1. Biesta, G.J.J. (2009) How to use pragmatism pragmatically: suggestions for the 21st century, *Education and Culture* 25(2), 34-45

2. Biesta, G. (2010) Pragmatism and the philosophical foundations of mixed methods research”, in *Sage Handbook of Mixed Methods in Social & Behavioral Research*, Tashakkori, A. and Teddlie, C. (eds), Sage, Thousand Oaks, CA, 95-117

3. Brandom, R. (ed) (2000) *Rorty and his Critics*, Blackwell Publishers, Malden, MA

4. Bygstad, B. & Munkvold, B.E. (2011) In search of mechanisms: conducting a critical realist data analysis, *Proceedings of the International Conference on Information Systems*, Shanghai, 10-13 Dec

5. Bygstad, B., Munkvold, B.E. & Volkoff, O. (2016) Identifying generative mechanisms through affordances: a framework for critical realist data analysis, *Journal of Information Technology*, 31(1), 83-96

6. DeForge, R. & Shaw, J. (2012) Back-and fore-grounding ontology: exploring the linkages between critical realism, pragmatism, and methodologies in health & rehabilitation sciences, *Nursing Inquiry*, 19(1), 83-95

7. Dewey, J (1931) *Philosophy and Civilization*, Minton, Balch & Co, New York, NY

8. Easterby-Smith, M., Thorpe, R. & Jackson, P.R. (2015) *Management and Business Research*, Sage, London

9. Feilzer, M.Y. (2010) Doing mixed methods research pragmatically: Implications for the rediscovery of pragmatism as a research paradigm, *Journal of Mixed Methods Research*, 4(1), 6-16

10. Festenstein, M. (2003) Politics and acquiescence in Rorty's pragmatism. *Theoria*, 50(101), 1-24

11. Fletcher, A.J. (2017) Applying critical realism in qualitative research: methodology meets method, *International Journal of Social Research Methodology*, 20(2), 181-194

12. Goldkuhl, G. (2012) Pragmatism vs interpretivism in qualitative information systems research, *European Journal of Information Systems*, 21(2), 135-146

13. Goles, T. & Hirschheim, R. (2000) The paradigm is dead, the paradigm is dead ... long live the paradigm: the legacy of Burrell and Morgan, *Omega*, 28, 249-268

14. Gomez, R. & Day, S.A. (2013) Research questions, paradigms and methods in ICT for development: content analysis of selected ICTD literature, 2000-2010, in *Into the Future: Themes Insights and Agendas for ICT4D Research and Practice*, N. Hayes & R.L. La Rovere (eds), IFIP WG9.4, Ocho Rios, Jamaica, 301-317

15. Heeks, R. & Ospina. A.V. (2019) Conceptualising the link between information systems and resilience: a developing country field study, *Information Systems Journal*, 29(1), 70-96
16. Heeks, R. & Wall, P.J. (2018) Critical realism and ICT4D research, *The Electronic Journal of Information Systems in Developing Countries*, 84(6), e12051

17. Henfridsson, O. & Bygstad, B. (2013) The generative mechanisms of digital infrastructure evolution, *MIS Quarterly*, 37(3), 907-931

18. Hevner, A.R. (2007) A three cycle view of design science research, *Scandinavian Journal of Information Systems*, 19(2), 87-92

19. Johnson, P. & Duberley, J. (2000) *Understanding Management Research*, Sage, London

20. Kadlec, A. (2006) Reconstructing Dewey: the philosophy of critical pragmatism, *Polity*, 38(4), 519-542

21. Mingers, J. (2002) Real-izing information systems, in *Proceedings of the 23rd International Conference on Information Systems*, L. Applegate, R. Galliers & J.I. DeGross (eds), Association for Information Systems, 295-303

22. Mingers, J. (2004) Re-establishing the real: critical realism and information systems, in *Social Theory and Philosophy for Information Systems*, J. Mingers & L. Willcocks (eds), John Wiley, Chichester, UK, 372-406

23. Morgan, D.L. (2007) Paradigms lost and pragmatism regained: methodological implications of combining qualitative and quantitative methods, *Journal of Mixed Methods Research*, 1(1), 48-76

24. Mungai, P. (2018) Causal mechanisms and institutionalisation of open government data in Kenya, *The Electronic Journal of Information Systems in Developing Countries*, 84(6), e12056

25. Norton, B.G. (1999) Pragmatism, adaptive management, and sustainability, *Environmental Values*, 8(4), 451-466

26. Ospina, A.V., Heeks, R., Ishida, L., Ssenkima, S., Mabirizi, G., Mugabi, N., Namarome, T. & Wandega, E. (2016) *Benchmarking Resilience of Agricultural Livelihoods*, Centre for Development Informatics, University of Manchester, UK

27. Peffers, K., Tuunanen, T., Rothenberger, M.A. & Chatterjee, S. (2007) A design science research methodology for information systems research, *Journal of Management Information Systems*, 24(3), 45-77

28. Ram, M., Edwards, P., Jones, T., Kiselinchev, A. & Muchenje, L. (2014) Getting your hands dirty: critical action research in a state agency, *Work, Employment & Society*, 29(3), 462-478

29. Reason, P. (2003) Pragmatist philosophy and action research: readings and conversation with Richard Rorty, *Action Research*, 1(1), 103-123

30. Saunders, M., Lewis, P. & Thornhill, A. (2016) *Research Methods for Business Students*, 7th edn, Pearson Education, Harlow, UK

31. Sousa, F.J. (2010) Meta-theories in research: positivism, postmodernism, and critical realism, in *Organizational Culture, Business-to-Business Relationships, and Interfirm Networks*, A.G. Woodside (ed), Emerald, Bingley, UK, 455-503

32. Thapa, D. & Omland, H.O. (2018) Four steps to identify mechanisms of ICT4D: A critical realism-based methodology, *The Electronic Journal of Information Systems in Developing Countries*, 84(6), e12054

33. Williams, C.K. & Karahanna, E. (2013) Causal explanation in the coordinating process: a critical realist case study of federated IT governance structures, *MIS Quarterly*, 37(3), 933-964
34. Winter, R. (1989) *Learning from Experience, Principles and Practice in Action Research*, Falmer Press, London

35. World Bank (2013) *Building Resilience*, World Bank, Washington, DC

36. Wynn Jr, D. & Williams, C.K. (2012) Principles for conducting critical realist case study research in information systems, *MIS Quarterly*, 36(3), 787-810