Source document: https://docs-lawep.s3.us-east-2.amazonaws.com/1696588739498.pdf (Vanuatu)
Community Based Contractor (CBC Evaluation) Assessment and Guidance Date of Report: 28 March 2018 Version No: Draft No.2 CBC Evaluation Report – March 2018 Version Control Version No.
Date Description Draft#1 8 Dec 2017 Draft#2 27 Mar 2018 First issue to DFAT (accepted by DFAT 12 Jan 2018, but on understanding that technical evaluation added) Second draft with technical and economic assessment of CBC model added to this evaluation report Acronyms Acronym CBC COPO DFAT DM FGD GoA GoV PWD R4D RME Definition Community Based Contract Community Partnership Officers Department of Foreign Affairs and Trade Divisional Manager Focus Group Discussion Government of Australia Government of Vanuatu Public Works Department Roads for Development Program Roads Maintenance Engineer 1 CBC Evaluation Report – March 2018 TABLE OF CONTENTS 1 Introduction .......................................................................................................................... 3 1.1 Evaluation Background ..............................................................................................................3 1.2 Operating Context - Vanuatu .....................................................................................................3 1.3 Operating Context – CBC ...........................................................................................................4 Purpose and Objectives of the Study .........................................................................................5 1.4 2 Methodology and Limitations ................................................................................................ 6 2.1 Methodology Design and Participants .......................................................................................6 Limitations of the Study .............................................................................................................7 2.2 3 Findings and Results – Part 1: Community Perceptions ........................................................... 9 Basic Profile ................................................................................................................................9 3.1 CBC Training Program ............................................................................................................. 10 3.2 The CBC Works Program ......................................................................................................... 11 3.3 CBC Funding Mechanism and Payments ................................................................................ 13 3.4 3.5 CBC Grievances and Community Complaints ......................................................................... 16 3.6 Overall Perception and Analysis ............................................................................................. 17 3.7 Qualitative Assessment of the CBC model.............................................................................. 17 Introduction ....................................................................................................................... 17 3.7.1 3.7.2 Communication and Engagement ..................................................................................... 18 3.7.3 Payment Process ............................................................................................................... 19 3.7.4 Grievances ......................................................................................................................... 20 3.7.5 Future Guidance on Improvements to CBC Model ........................................................... 21 4 Findings and Results – Part 2: Technical Assessment and Analysis .......................................... 22 Types of Works ....................................................................................................................... 22 4.1 Technical Assessment of the Appropriateness of CBC Works ................................................ 22 4.2 4.3 Quality Comparison of Different Grass-Cutting Methods ...................................................... 26 4.4 Comparison of Cost-Effectiveness .......................................................................................... 27 4.5 Other Considerations .............................................................................................................. 31 4.5.1 Procurement and Management Arrangements ................................................................ 31 4.5.2 Community Performance and Relationships ..................................................................... 32 4.5.3 Health and Safety of CBC Workers and Road Users .......................................................... 32 Increasing CBC Scope of Works ......................................................................................... 33 4.5.4 5 Conclusions and Recommendations ...................................................................................... 35 Annex 1 CBC Survey Tool – Quantitative Survey ........................................................................... 39 2 CBC Evaluation Report – March 2018 1 INTRODUCTION 1.1 Evaluation Background The Roads for Development (R4D) Program is a A$32.4 million, five-year program funded by the Government of Australia (GoA), through the Department of Foreign Affairs and Trade (DFAT), to assist the people of Vanuatu to gain increased access to a well maintained rural road network.
It supports the Government of Vanuatu (GoV) to more effectively plan, construct and maintain its road transport infrastructure.

The A$32.4 million was split between a grant made available to the Public Works Department through a Direct Financing Agreement (DFA) and technical assistance funded directly by GoA.
The R4D program aims to streamline the Public Works Department’s management and administration and provide institutional strengthening support for its strategic transition from undertaking works directly to managing more efficient outsourced service contracts.

The program therefore stimulates and develops the low private sector base and provides technical engineering support to improve sustainable access in the medium to long term leading to economic and social benefits for Vanuatu and its communities.
This last phase of the current program also directly involves island communities in maintaining roads, thus stimulating local economic activity whilst also improving road condition and accessibility.

As part of its commitment to deliver quality road infrastructure in rural Vanuatu, PWD has sought a number of key strategies to engage with local businesses and communities.
The purpose is to deliver roads in a cost effective manner while at the same time maintaining the long-term viability and sustainability of the network.

The engagement of communities through community-based contracts (CBCs) is an important component of overall routine maintenance of the rural road network across Vanuatu.
The primary purpose of this evaluation is to assess the effectiveness of the CBC model as a means of engagement with communities and to consider whether the model should be further extended across Vanuatu.
The evaluation also covers an assessment of technical aspects related to the application of CBC model, including (i) quality of works; (ii) cost-effectiveness; (iii) effectiveness of procurement, supervision and administration of CBC projects; (iv) sustainability; (v) safety of workers and road users; (vi) flexibility of the model to changes in the method of works delivery; (vii) flexibility of the model to changes in the contracting arrangements; and (viii) alignment of CBC model with broader departmental objectives.
The comparative assessments were made using theoretical and existing contracting models and works methods, in order to establish the circumstances under which each of them gives the most benefit from Public Works Department’s perspective as it transitions towards becoming a Road Network Manager.

A secondary objective of the evaluation is to compile lessons and recommendations that can inform and shape R4D’s and PWD’s future investments in community-based engagement strategies.
1.2 Operating Context - Vanuatu Vanuatu is an archipelago nation consisting of over 80 widely scattered large and smaller populated islands.

This makes logistics and management of equipment-based roadworks programmes (and the use of larger scale construction equipment) problematic.

Climatically Vanuatu has very high seasonal rainfall intensities and runoff, so that serious flash flooding, which causes extensive damage to roads, is a regular but unpredictable event.
Vulnerability to effects of climate change is high and events are expected to become more frequent and intense.

In Vanuatu, climate is the biggest factor affecting deterioration of roads, not traffic.

The country has a small total population and it is highly dispersed with around 76% of its people living in numerous small village settlements spread over its many islands.
There are significant social and 3 CBC Evaluation Report – March 2018 cultural differences across the country, with an extreme degree of diversity in language and customs: 112 languages in a population of only 260,000.

Language and tribal groupings are the primary sources of identity and belonging in Vanuatu.
The presence of people from outside these groups within any local area is often met with suspicion unless formal traditional protocols of respect are followed with local leaders (Chiefs) and permission is formally given to enter and carry out any activity within a given area.
Like many Pacific nations and small island states, Vanuatu continues to struggle to maintain its road system due to under-investment, limited preventive and periodic maintenance, and inefficient repair and maintenance models that rely too much on centralised management and force account capacity.
The country faces some serious challenges in provision and maintenance of its transport infrastructure: a small formal economy with a low revenue base; a small technical skills pool and a small, ill equipped and underfunded PWD to deliver its roadworks.

As a result, the implementation of the CBC model was viewed as a low-cost means to provide income to local communities while supporting basic road maintenance.
1.3 Operating Context – CBC Routine road maintenance is an integral component of any road asset management strategy.

In Vanuatu, maintenance is often under-prioritised in favour of the establishment of new roads, expansion of the overall network and improvement/upgrading projects, which often gain much greater political support.

Through R4D support and encouragement, PWD now places a much higher priority on routine road maintenance.
To support ongoing maintenance and especially routine maintenance, which consists mostly of grass and vegetation cutting, drain clearance and gravel road pothole repairs, a CBC model was proposed in late 2014.

The CBC model commenced its trial in 2015 with the establishment of a CBC Framework.

Key components of the trial were considered, prioritized and agreed as part of the framework.

Additional staff were recruited to support the concept and model.
Additional staff included: a Senior Community Partnership Officer (SCPO) and three Provincial CPOs.

Their priority task was to establish partnerships with government at the provincial and district level.

Through this engagement, the CPO’s engaged and coordinated with communities (through recognized traditional authority structures which were the foundation of the CBC model) to formally engage them to support road maintenance works.

In mid-2015, contracting approaches and modalities were finalised.
The initial concept was for CBCs to be selected according to each island’s traditional governance structures to conform to the most cohesive social/governance groupings.

This principle was to align contract lengths as closely as possible with community-recognised boundaries to prevent land disputes due to sensitivities around land ownership and boundary.
The CBC model also sought to establish a stable and consistent system of contracts, that once established will not change much in terms of contracting entities and contract lengths from year to year, thus reducing contract administration tasks, and making it easier to re- establish contracts at the beginning of each year.
This CBC program is aligned with the GoV’s Decentralization Act 1994 (CAP230) and supports decentralization through engagement with provincial administrations, Area Councils, Councils of Chiefs and communities.

An additional purpose of the CBC model is to build community capacity so they can develop their communities using some of the funds from these contracts.
In supporting this process, R4D has also introduced a series of social and environmental safeguards which provide a basis for sustainable ongoing capacity support and development to communities.

In the 2018 Workplan, all CBC contracts are funded entirely by PWD as R4D grant funding will cease at the end of June 2018.

However, R4D funding for CPOs and other support of the program, such as provision of safety equipment will continue through to the end of June 2018.
4 CBC Evaluation Report – March 2018 1.4 Purpose and Objectives of the Study The evaluation focused on the effectiveness and efficiency of the PWD/R4D CBC investments and based upon the findings suggests guidance on possible future adjustments to the CBC model which may be considered to ensure the long-term sustainability.
The evaluation was therefore designed to achieve the following:     to outline the relevance of the CBC model in engaging communities to participate in a positive and proactive manner to support road maintenance, to evaluate the effectiveness of the model in achieving the above, to evaluate the technical elements of work and to provide a comparative analysis of the CBC model against other delivery options and models, including a comparison of costs to assess the success of the model in terms of distributing income to communities and looking at how that income is being prioritised and utilised.
The evaluation provides key recommendations and guidance based on the findings and analysis to inform future R4D and PWD investments in CBC contracts.

The evaluation had two main evaluation questions to address.

They were:  To what extent did the CBC program provide an effective model to promote engagement and ownership from communities?

 To what extent is the current payment mechanism functioning in terms of allocating resources and promoting community engagement?
Annex 1 provides a more detailed outline of the evaluation questions used during both the quantitative and qualitative components of this evaluation study.
5 CBC Evaluation Report – March 2018 2 METHODOLOGY AND LIMITATIONS 2.1 Methodology Design and Participants The evaluation applied a convergent parallel design methodology.1 The approach entailed the implementation of a quantitative survey across all CBC contracts which were supported by a smaller sample of qualitative research looking at specific areas to confirm and validate survey findings.

The qualitative component employed a case study format informed by a focus group discussion (FGD) methodology.
The review commenced with a desktop review of available documentation related to the CBC model.

The evaluation also considered other community based models applied in other development contexts with a specific emphasis placed on models within the Pacific region.

As part of the evaluation design, the program engaged an appropriate sampling frame for the completion of the study.

The methodology for the evaluation applied a mixed methods approach.
The rationale for this selection was that the reliance on one method may not generate the depth of findings required to make definitive assessments around effectiveness and efficiency.

Importantly, the triangulation of results is integral in a study like this to remove aspects of subjectivity, bias and an over reliance on one set of data.
Importantly, the evaluation strongly emphasised a utilisation-focused evaluation in that results and issues will be analysed and presented in a way that informs decision-making and provides a sound evidence base for decisions.

For the quantitative study, a survey was uploaded to tablets to enable ease of use and collection of data.

All CBC contracts were surveyed as part of the quantitative component.
A selected sample of 20 CBC’s contracts were then purposefully selected across all provinces to participate in more in-depth FGD’s with the intention to seek a wide cross-section of detailed opinions regarding the CBC model.

The study also included a technical assessment of the model.
This was done by following: (i) analysis of the information obtained from various documentation; (ii) consultations with PWD/R4D personnel involved in the management of CBC works; (iii) consultations with personnel of other contracting entities and relevant PWD/R4D personnel; (iv) use of the factual knowledge and available data, and (v) the comparative assessment of works outputs and comparison with other contracting models.
For data processing and analysis, team members reviewed the responses to the FGD questions and developed simple scoring frameworks which were used to assist in developing findings and to identify key themes and issues.

All findings were then consolidated and peer reviewed through internal team discussions to ensure all the key points were adequately and properly reported.

Following the analysis, small case studies were prepared documenting the key findings.
Summaries of the overall findings are included in this report.

A total of 143 CBCs were interviewed during the quantitative study.

A total of 19 CBCs were selected to participate in the FGD’s.

The table below summarises the breakdown of CBCs included in the quantitative evaluation.
1 In this model, the evaluator collects and analyses quantitative and qualitative data separately on the same subject and then the different results are converged (by comparing and contrasting the different results) during the interpretation.

Evaluators use this model when they want to compare results or to validate, confirm, or corroborate quantitative results with qualitative findings.
The purpose of this model is to end up with valid and well-substantiated conclusions about a single subject (Creswell, 2014).

6 CBC Evaluation Report – March 2018 ISLAND PROVINCE TOTAL CBC TO BE INTERVIEWED COMPLETED NO RESULT TANNA SANTO TAFEA SANMA AMBAE (NORTH & EAST) PENAMA MALEKULA MALAMPA EFATE SHEFA TOTAL 25 46 39 22 26 158 Table 1 : Evaluation Participants and Engagement 23 45 39 21 15 143 2 1 0 1 11 15 Interviews for the evaluation were planned over an eight-week period (May-June 2017).
There were delays involved due to weather conditions and access to some islands.

Also time was required to notify all CBCs and to ensure they were available for interviews.

No changes were made to the methodology or approach from the baseline.

This ensured a level of consistency across the data collection points.

The technical analysis was completed in early 2018 following a request for additional information and data to support the initial evaluation work with communities and CBCs.
2.2 Limitations of the Study All evaluations and reviews have limitations.

This evaluation did have some possible limitations that had to be managed as part of the overall process.

Key limitations for the study included:  Lack of reliable records: which restricted the scope of the evaluation to the recent history.

Many of the findings are based on experience, instead of relying on systematically collected records which would have provided a more objective approach.
 Time and Resources: the rigour of the data gathering analysis was constrained to some degree by the time available.

The evaluation team were unable to spend considerable time with communities, particularly when follow-up meetings and discussions were required.

 Experience of the CPOs: this was the first attempt to collect data using electronic tablets and the first time the CPOs had collected the data.
Several CPO’s could not complete all community visits in the timeframe available and there appeared to be a number of issues associated with completion of data collection and accurate recording.

 Training and awareness raising: this was probably the largest limitation in that there was an expectation that CPOs had the capacity and knowledge of relevant communities and also the time to complete the surveys.

A key lesson has been learned for next time.


 Access to work sites/communities: travel to the field for data collection was impeded in some cases by weather, availability of stakeholders and time constraints.

Not all CBC’s were willing or able to participate in the study.
 Subjective observations from the consulted parties: some of the views may have been biased or contradictory to the views of others, and it is thought that some evaluators may have mitigated this by aggregating the views of participants, to an unknown extent.

7  Comparative assessment in certain cases lacked the basis for comparison: since some technical aspects were practised only through the CBC model.
Theoretical models were created from the factual and derived data to allow for comparative assessments to be made.
CBC Evaluation Report – March 2018 8 CBC Evaluation Report – March 2018 3 FINDINGS AND RESULTS – PART 1: COMMUNITY PERCEPTIONS The following sections provide an outline and summary of key analysis related to the findings generated from both the quantitative and qualitative aspects of the study related to community perceptions and feedback.2 This aspect of the study involved an initial quantitative survey distributed to all CBCs and a follow-up qualitative based focus group discussion process was facilitated in a smaller sample of communities.
The second part of the evaluation which follows examined the quality of CBC works by assessing if routine maintenance works contribute positively to the preservation of the asset and to what extent.

It compares how potential different modes of routine maintenance delivery compare regarding quality and costs The findings presented in this report were presented at a recent PWD Annual Planning event in November 2017 for information and further discussion.
3.1 Basic Profile It would be expected that all CBC’s have a finalised contract.

The evidence suggested in the graph below is that all CBCs do have an executed contract in place.

Ambae is an outlier and there may been some misunderstanding of the question given.

According to the findings, 92% of all CBC’s on the island have no contract.

This is actually impossible as payments cannot be made to CBCs without an executed contract in place.
However, the general finding is that all CBCs have a contract in place.

50 40 30 20 10 0 Efate Santo Malekula Yes No Tanna Ambae Graph 1: Does your community have a CBC contract?

The initial hypothesis of the study was that the CBC model has been very popular amongst communities as a means of engaging with the road network near to their community and in providing an opportunity for income generation and community involvement.

Communication appears to be a major issue between CPOs and communities.
Several communities indicated that they had little or no understanding of the terms of their contract and that is had not been explained clearly.

It is also evident from interviews with the CBC team in PWD HQ that there have been recent transitions from old contracts under a former similar program to the current CBC model, and that transition is still ongoing in some areas.
This may have resulted in some initial confusion over contracts, content and 2 It is noted that some aspects of the data do not appear accurate or correct.

These issues were raised at the PWD Annual Planning Workshop and will be followed up by the CBC team.

For the purposes of this evaluation data is presented as it was received so as to maintain high standards of accountability and transparency.

This footnote is made purely for contextual understanding only.
9 CBC Evaluation Report – March 2018 expectations.

There is a strong need for CPOs to be trained and familiarised with contract knowledge to best support the CBCs in contract understanding and administration.

In asking communities how they had heard about the CBC concept, Graph 2 below provides an outline of the results collected.
It appears that PWD has done well through its divisional network to promote the benefits of the CBC concept with most communities reporting across each of the provinces that PWD is the main source of information.

In Ambae, the data suggests a greater focus on community leaders (which could also in turn mean PWD provided the initial message to the Community Leaders).

50 45 40 35 30 25 20 15 10 5 0 Efate Santo Malekula Tanna Ambae Community Leader PWD Prov/Dist.
Govt Neighbouring Community Other Graph 2: How did your community hear about the CBC concept?

Controlling information about the CBC contracting model may become important from PWD’s perspective as more communities seek to engage on CBC contracts moving forward.

PWD need to have clear guidelines to all stakeholders (provincial officers and communities) on dissemination of information to avoid overburden of the system with too many CBC contracts.
Overall the provision of works has been well received by communities as they are suffering somewhat given the shifting of young people to Port Vila and Luganville.

This urban drift has resulted in less rural income opportunities.

3.2 CBC Training Program There is a requirement for all CBCs to have received training on PWD’s social and environmental safeguards before commencing work on site.

This is supposedly a mandatory process.


However, when asked if CBC’s have received training there were quite a number of CBCs (particularly on Efate and Santo) that indicated they have not received any training.

Of greater concern was the general spread of CBC’s across all provinces that either did not receive training or could not recall the training.

Graph 3 provides a summary of the findings.


In following up the finding with the CBC team in PWD HQ, questions were raised as to whether or not some CBCs might have participated in some other training.

The finding highlights an urgent need to critically review the training program currently implemented by the CPOs and to ensure that expectations are clear.
Considerable resources have been invested into the development of social and environmental safeguards and workplace health and safety manuals which appear to have not been fully distributed to CBCs according to the latest findings.

10 CBC Evaluation Report – March 2018 50 40 30 20 10 0 Efate Santo Malekula Tanna Ambae Yes No Don't Know Graph 3: Has the community received road maintenance training (3 modules through PWD)?
A question on the type of CBC work communities undertook (grass cutting, pruning, tree planting etc.)

was included in the questionnaire.

However, for some reason, CPOs appeared to skip over this question.

There wasn’t significant data provided to adequately map key responses.

In terms of training for the future the following graph provides an outline of the preferred topics from CBC’s by province.
50 40 30 20 10 0 Efate Santo Malekula Tanna Ambae Technical Training Safety First Aid Financial Literacy Gender and Disability No Training Graph 4: Additional Training Priorities The results do indicate some interesting findings in relation to technical training and safety.

This possibly indicates that CBCs have received training at some point as they are cognisant enough to request training in safety.
Results from the FGDs also indicate that most CBCs would like to move beyond simple grass cutting and vegetation clearance/pruning.

The analysis of these findings is provided later in the report.

Of interest, was the high preference for first aid training in Ambae.

The CBC team is following up with the CPO from Ambae to confirm the accuracy of the data.
3.3 The CBC Works Program A disappointing result from the evaluation was the missed opportunity to obtain deeper insights into the type of work CBCs are currently involved with.

The main purpose of the CBCs were originally to support basic road maintenance to complement the work of national contractors, IBCs and Force Account Teams.

This involved a mix of roadside clearing (grass, bush and drainage) as well as pothole 11 CBC Evaluation Report – March 2018 patching on gravel roads.
Over time, the focus of work appears to have shifted to simple grass cutting and branch clearing only, although there are plans to address this in 2018 with the re-introduction of pothole patching by some communities.

As a result of this shift, it has become more challenging to assess the actual contribution of communities towards sustainable maintenance of the road network.
R4D’s Road Operation Specialists (ROSs) undertook a desk review based upon their knowledge of CBC works undertaken, to assess the adequacy3 and quality of works completed by CBCs by each province.

Graphs 5 and 6 below provide an outline of the assessment based on professional judgement and knowledge of the work areas.

80 60 40 20 0 Penama Sanma Malampa Tafea Adequate Not adequate Graph 5: Adequacy/Appropriateness of CBC Work Work appears to be assessed as most adequate in Malampa and Tafea.
The high amount of work classed as non-adequate in Penama and Sanma requires further investigation during 2018.

70 60 50 40 30 20 10 0 Penama Sanma Malampa Tafea Good Fair Poor Graph 6: Quality of the Work The overall indication is that the best work is occurring in Malampa province.

Both Penama and Sanma have mixed results and require further investigation.

3 “Adequacy” refers to whether the work is adequate or not – i.e.

whether it made sense to do the work in that particular location.
It might have been better to use the term appropriateness.

For instance, we may have situations where grass cutting is undertaken on an un- engineered, un-drained road where it is not required.

“Quality” refers to the ROS’s subjective view of the quality of work undertaken.

12 CBC Evaluation Report – March 2018 3.4 CBC Funding Mechanism and Payments Over 95% of all respondents were aware that PWD had made funding available to support the implementation of the CBC model.
All communities appear to have established some form of working group for the collection and use of funds.

One interesting note has been the rise of the individual or household contracts for CBC’s.

It is unclear from the results whether there was any disaggregation of results but it does appear that irrespective of whether the CBC was a household or a community that the answers were discussed and agreed prior to providing a response.
In terms of financial management, traditional structures of chiefs and churches tend to dominate the collection and management of funds for CBC works.

One interesting note is on Santo; one IBC has taken responsibility for the handling of funds for a number of CBCs.

In addition, there is limited distinction between a chief and chairman in many communities, where the individual could be the same person.
However, the report covers all the responses provided, and the results suggest there is a distinction in some communities.

50 40 30 20 10 0 Efate Santo Malekula Tanna Ambae Chief Church Senior Community/Board Members Chariman Contractor Graph 7: Who collects and handles the funds?

When asked how funds had been spent, the following graph summarises the responses from communities.
25 20 15 10 5 0 Efate Santo Malekula Tanna Ambae Shared with Community Church Community Infrastructure Wages and Salaries Community Activities School Fees Graph 8: How have CBC funds been spent?

Not surprisingly, communities tend to spend money received on community resources and infrastructure.
Whereas Island-based Contractors (IBCs) tend to retain profits and reinvest into their 13 CBC Evaluation Report – March 2018 business, communities tend to focus on the wider collective group and seek to contribute towards public buildings and school fees.
Unlike other responses in the questionnaire, the findings tend to be relatively evenly distributed across a range of responses, except in Ambae where the church and community infrastructure dominate, and Tanna where wages/salaries and school fees dominate.
The qualitative focus groups also confirmed the data with representative communities indicating that the CBC model is another source of income for the community and that joint decisions are often made early with regards to how funds are spent and utilised.

For example, a community in Penama indicated the following: Currently the community will be saving their payments for this contract as they are yet to decide on what community project they would like to put the money towards.
Previously, the payments have gone to individual households and not shared.

The community are pleased that this is a sustainable program and they have the autonomy to decide how the payments can benefit the community.

The community can therefore create a plan of further improvements.

Village Elder, Bangaelo, Ambae, Penama Province The greatest challenge of the CBC model appears to be the timely payment of funds.
Whilst communities are generally accepting of the model in theory the practical application and payment for work is a consistent theme of concern amongst communities.

From the feedback received, communities are waiting approximately 2 months before funds are received.

40 35 30 25 20 15 10 5 0 Very Happy Happy Neutral Efate Santo Malekula Tanna Ambae Graph 9: Satisfaction with way funds are handled and distributed.
In terms of the funds being distributed and put to good use, most communities surveyed were very happy with the outcome.

14 CBC Evaluation Report – March 2018 40 35 30 25 20 15 10 5 0 A lot Somewhat Neutral Not good Worse off Efate Santo Malekula Tanna Ambae Graph 10: Has the money improved the community Most communities surveyed have a high regard for the CBC program and believe that it has had a positive contribution towards making the community better.
One positive outcome has been a growing sense of community engagement, not only amongst the communities themselves but between PWD divisional offices and communities.

Graph 11 provides a summary of this data.

50 40 30 20 10 0 Efate Santo Malekula Tanna Ambae Yes No Don't Know Graph 12: Is the CBC model contributing to better engagement?

Of concern are the reports that while engagement has increased there are still some concerns over the role of the CPOs.
Many communities from the focus groups have requested more consultation and awareness from PWD.

One community from Penama indicated that “there has been little to no consultation with the communities by the CPOs.

This has led to communities believing that PWD does not want to engage and the community does not know where to go when there is an issue.” The level of engagement does correlate positively with the community perception that they are the “owners” and custodians of the road.
15 CBC Evaluation Report – March 2018 40 30 20 10 0 Very Much Somewhat Neutral Not sure Not at all Efate Santo Malekula Tanna Ambae Graph 13: Community perception as owners of the road The community maintains a strong sense of ownership of the road which is a positive outcome in terms of longer-term sustainability and engagement.

Ownership perceptions are consistent across all islands and provides a basis for further engagement moving forward based on the work of the CBC model to date.
3.5 CBC Grievances and Community Complaints All community based programs experience some level of disruption, dispute or grievance.

The best way to prepare for inevitable disputes or grievances is to maintain a robust and transparent grievance mechanism which is promoted, discussed and applied.
Through the social safeguards framework and development of associated manuals, R4D has supported PWD with the development of a robust grievance mechanism aimed at identifying issues, working with communities and also to provide relevant and practical solutions within an appropriate timeframe.

Graph 14 below provides no major surprises in that communities are reporting some level of grievance.

The major concern is the skewed results for Tanna and Ambae.
For Tanna, there have been numerous disputes and grievances over the past 2-years which have been addressed so it is surprising to see a majority of communities reporting no grievance.

The results for Ambae also appear unrealistically skewed.

These results indicate some miscalculation or misrepresentation by the CPOs when collecting data for this evaluation.
25 20 15 10 5 0 Efate Santo Malekula Tanna Ambae Yes No Don't Know Graph 14: Incidence of Grievances 16 CBC Evaluation Report – March 2018 Of greater concern is a general lack of awareness of PWD’s formal grievance mechanism.

Like the CBC training mentioned above, the development of a grievance mechanism is an important part of the CBC model.

The data suggests that many communities across Vanuatu have reported that they are not aware of the formal mechanism.
CPOs have been charged with informing communities of the grievance mechanism and to promote closer working relationships between PWD and communities, so this data is problematic and requires resolution.
50 40 30 20 10 0 Eftae Santo Malekula Tanna Ambae Yes No Don't Know Graph 15: Awareness of PWD’s formal grievance mechanism 3.6 Overall Perception and Analysis The overall perception of the CBC model from a community perspective is that it is an excellent model through which to engage and involve communities in road maintenance activities which have the secondary advantage of community-level income generation.
50 40 30 20 10 0 Efate Santo Malekula Tanna Ambae Excellent Good Neutral Not Good Poor Graph 16: Overall impression of CBC model 3.7 Qualitative Assessment of the CBC model 3.7.1 Introduction The following analysis section draws upon the qualitative assessment of the CBC model and draws out a number of other key findings that support the quantitative data and information provided in the earlier sections.

Qualitative data was collected through a sample of focus groups.
The analysis provided provides further evidence and guidance to support the key conclusions and recommendations moving forward.

17 CBC Evaluation Report – March 2018 The most positive aspect of the CBC model is surprisingly not about financial gain or employment, but about communities having a stronger sense of engagement, commitment and involvement.

All FGD’s across the islands reported enhanced community relationships and welcomed the opportunity to discuss and plan events and spending priorities.
The CBC model has provided a basis for joint decision making and additional investment in local community infrastructure.

As indicated in the quantitative analysis, community infrastructure was the highest priority area in which to invest funds as opposed to individual household or personal savings or expenditure.

An interesting case was from Managalilou village in Efate where the women spoke of striving for financial independence.
The text box provides a summary of the story shared by a female member of the community.

Women of two church groups wanted to travel to Fiji on an overseas fact-finding trip.

They first asked their husbands if they could have money for the trip.

When the husbands declined, they went to the chief to ask to be part of the grass cutting program to earn the money for the trip.

The chief, Tobin, allowed the women to take part in the grass cutting program on all four cuts.
The money that was raised was used by 35 women to travel to Fiji to research how Fijian women ran and operated their markets.

The women wanted to see how Fijian women ran their businesses so they could expand their own road side market and branch out into a road side food stall.
(Managalilou, Efate) Alternative employment was also identified as an effective outcome of the CBC model in that some communities (often in non-agricultural lands or with limited access to markets) could work and earn a small income.

The ability to earn income was rated highly in addition to associated increases in morale, engagement and community involvement.

3.7.2 Communication and Engagement Communication has certainly improved as identified through the initial survey and the follow up FGDs.
Across all communities in the FGDs there was strong internal communication and evidence of joint decision-making.

Communities are generally seeking new and improved ways to promote a shared understanding of requirements and outlining of key responsibilities within the community.

In Sanma, the Tano Voli community provides a useful study and example of strong community structures.

As a community, they decide how the payments will be divided.
They have allocated a small sum of money from the payments received to be invested into work equipment and fuel.

The Chairman, CWSs and community members who cut the grass are paid a small salary for their work.

The remaining funds have been put towards a new water pump to give running water to the community.

The community prior to this did not have running water.

A small amount will also go to community welfare, supporting households which require help with payment of school fees, clothing and food.
The community has also expressed a desire to be involved with training prior to commencing works.

This would build confidence within the community and help with ensuring safety for the community working on the road.

The cycles of payments are divided between the men and the women.

The women take the first cycle and the men take the second cycle.
They have achieved much of what R4D/PWD would like the community to do: investing in work equipment; working collaboratively to decide on community projects; investing in community improvement; transparency with distribution of funds including a small wage for workers, CWSs and the Chairman.

Another example quote from discussions in Sanma is given below: 18 CBC Evaluation Report – March 2018 The community is happy to be involved with the CBC program.
This community has focused on new skills and capacity building for women, youth and people with a disability.

From participating in the program, everyone is included which strengthens the community.

This community has one person with a disability participating in grass cutting.

The community stated "Everyone can be involved; it doesn't matter".

There is a strong sense of inclusion with this community.

People have had the chance to learn how to use work equipment, such as grass cutters and lawn mowers.
(Valankara) In moving forward there is a need to engage not only just the chief and community leaders but with communities as whole.

Obviously there is a need to work through proper formalities and ‘kastom’ protocols.

However, there has been a request from many of the FGDs for PWD and PCOs to hold more community based meetings and discussions, either as part of a grievance mechanism or general report and feedback sessions.
A serious concern raised by all communities is that too often CPOs are not engaging in any formal community discussions.

Many communities expressed confusion over current contract arrangements and expectations.

In some cases, CPOs just drive by to “inspect works” without leaving the vehicle.

The CPOs are meant to be fully engaged with communities to address questions and concerns and provide a necessary link between communities and PWD.
At present the evidence suggests that all CPOs are not fully complying with roles and expectations and in some cases are causing damage to the reputation of PWD (and possibly DFAT as a donor to the program).
Following the evaluation, there has been little opportunity to speak to the CPOs directly but a recommendation arising from this report is for an immediate review and possible re-drafting of job descriptions and training of CPOs to reinforce and strengthen roles, responsibilities and behavioural expectations.

This is to be addressed as part of the R4D HR Specialist’s institutional reform activity.

Other challenges reported across the group include the occasional clash with other community programs.
For example, grass cutting exercises sometimes clashed with church programs, but again, this could be avoided by closer management and co-ordination by CPOs.

3.7.3 Payment Process The payment process is probably the most controversial aspect of the CBC system.

The process is fraught with continual delays, uncertainty and a general lack of transparency.
There are a number of contributing factors that influence the CBC’s perception of the payment process:  Lack of clarity and understanding of financial expectations, requirements and responsibilities  Limited explanation of the setting of work unit rates and the method of measurement to be applied  Limited follow-up of respective CPOs during grass-cutting and other CBC works  Limited support provided to prepare the necessary documentation (i.e.
signed measurement sheets)  Delays in the preparation of payment certificates and approvals by the Divisional Managers and their teams  Significant delays with processing after transfer of the payment request to PWD HQ with paperwork often referred back to DM’s and CPOs if submissions are incorrect or incomplete.

The contract requires 30-day payment terms, but this is not generally being achieved by PWD.

This contributes significantly to community tensions.
In some cases, the delay in payments works against the wider objectives of the program.

Through the FGD, some communities have expressed a desire to purchase equipment such as mechanical cutters and lawn mowers.

However, investment decisions are being stalled by communities due to uncertainty around payment dates.

19 CBC Evaluation Report – March 2018 There is also a feeling amongst some communities that payments should be linked to the varying degree of difficulty of the work.
All communities indicated that the first cut of grass each year was the most difficult.

In Sanma, a number of communities have shared concerns around transparency of the payment process, with confusion over why delays are occurring and where the payment is in the payment process cycle.

As indicated in one community, morale is now affected by late payments to the CBC: “The community does not understand why there is such pressure to do the tasks whilst not being paid for the previous task.
This has created low morale and can make it hard for the community to get people to work on the next round of road maintenance” (Community elder, Valankara).
A review of the payment process including (i) steps and people involved, (ii) the design of standard forms involved, (iii) approval authority levels/delegated authority levels, (iv) resource capacity to deal with the volume of payment requests, and (v) the role of PWD and MOF finance systems in the process, needs to be undertaken to identify blockages and implement streamlining and reforms.
It is acknowledged that there are some limitations and constraints around the full devolution of responsibilities to divisions.

However, a structured approach to streamlining payments would be appropriate, particularly to ensure a reduction in the processing time for payments (currently often greater than 30 days).

3.7.4 Grievances All community based programs have some level of grievance.
This is particularly evident in provinces with high levels of land ownership and particularly kastom and public lands.

The FGD’s revealed that all communities across the country are becoming more frustrated with PWD (in particular CPOs) due to lack of engagement and perceived lack of understanding of the needs of communities and their CBC’s.

There are some significant concerns relating to some CPOs.

The interview transcripts do provide significant detail.
Some serious questions have been realised around the capacity, capability and willingness of CPOs to play a role in effective community development.

Most communities perceive that there has been little engagement with PWD, in particular the CPO.

In one example, a CBC made the comment that the CPO will ride pass and not stop to talk with them.

They feel that their issues regarding their contract have not been heard.
There has not been enough consultation or relationship building with the community members who are part of the grass cutting team.

CBC’s need to feel engaged, supported and that their contributions to the road network are valued, but at a basic level simply want to know if they are undertaking their works as expected.

Besides the lack of perceived engagement most communities have raised the issue that the rates of pay are not commensurate with the level of work completed.
For example, all CBC’s report that the first cut of grass each year is the most difficult but that the pay for that task is the same as other cycles.

CBC’s have requested a range of scales applied to different categories of work and intensity.

This requires improved planning from PWD and a revision to guidelines, scopes of work and payment rates.

Lack of safety equipment and time taken to travel from villages to work sites is another common grievance.
For instance, Managalilou Village stated, “it takes the women a long time to travel to and from the work site and their village, travelling back after a whole day cutting grass.

The distance from their village to the road is 3kms.

The road from the Efate Ring Road to their village, until recently last month, was in extremely poor condition.

This has made it hazardous for the women to be walking back to the village when it is dark.
This impacts on tasks required to run their homes, such as cooking and childcare, as all the women manage the home”.

As there has been no partnership establishment process in some provinces, communities have not been invited to official launches of the CBC program, where safety equipment is handed over to each CBC.

This has meant that some communities have been working without safety equipment, such as 20 CBC Evaluation Report – March 2018 those on Efate.
Communities are very concerned about the dangers of working on the road and feel vulnerable without proper safety equipment.

This requires urgent attention, and is being addressed at the time of writing this report.

Safety equipment was handed over to CBCs on Efate on 24 November 2017.

Many of the problems raised could be addressed quickly through better engagement.

The PWD CBC team need to urgently review community engagement guidelines and practices and all CPOs must participate in refresher training.
This would also provide an opportunity for CPOs to present their own cases as it is also clear that some communities can be quite demanding and have high expectations.

Therefore, the workshop and meeting should be an opportunity to clarify positions, identify areas of good practices and see strategies to remove underperformance, and ultimately improve the effectiveness of the CBC program.
3.7.5 Future Guidance on Improvements to CBC Model Although this evaluation is a review of the CBC model and its effectiveness there is scope to provide guidance for consideration based on the results of the FGDs.

A number of key suggestions provided include:   consider allowing different categories of work in CBC contracts, and aligning payment rates to reflect the scope and difficulty of work; seek to increase the length of road to maintain under each contract or increase the scope of works.
At present, lengths and scopes on some contracts are shrinking and leading to increased community tensions.

The preference is to have at least 6km of road length to maintain per contract.

For example, in Lamlu, Tafea Province, they would like to have other tasks added to their contract to earn more money.

They would like to do pothole patching.


If PWD could provide gravel along with appropriate technical training and equipment, they would be happy to have this added to their contract;  Some CBCs have requested more gender sensitive considerations and flexibility.

Much of the work falls to women who also have other family responsibilities, and this is placing stress and pressure on women to meet expectations both at home and in the workplace.
In reviewing the analysis, it is clear that the CBC model needs to re-focus on its original goals and strategic intent, which was to deliver high-quality routine maintenance to roads at affordable prices.

The model is now seen by many communities as a model of convenience which provides small cash grants to communities.
However, the model itself is not working as well as it could, and is suffering from inefficiencies, poor engagement, inappropriate works being undertaken, poor quality of work (in some areas) and a greater number of complaints and grievances from CBCs.

There are, however, some good examples of the CBC model.

For instance, in Tano Voli, the community works very well with the CBC program.
They have achieved much of what R4D/PWD would like the community to do: investing in work equipment; working collaboratively to decide on community projects; investing in community improvement; transparency with payment amounts; allocating funds towards a small wage for workers, CWSs and the chairman.

With training and capacity building, this community, and others like it, can continue to actively engage in supporting the maintenance of the road network.
21 CBC Evaluation Report – March 2018 4 FINDINGS AND RESULTS – PART 2: TECHNICAL ASSESSMENT AND ANALYSIS 4.1 Types of Works Works normally undertaken by CBCs are routine maintenance works and include the following: (i) grass cutting; (ii) tree pruning; (iii) clearing of drainage structures and ditches.
Analysis of the CBC projects programmed within the 2018 workplan shows that around 91% of the total value of CBC works is assigned to grass-cutting activities, so this is by far the dominant activity undertaken by CBCs.
7,223,748 , 9% 74,820,168 , 91% Grass cutting Other works Graph 17: Value of grass-cutting works within the overall scope of CBC works 2018 Because the value of other types of works undertaken by CBCs is negligible, and consistent data on other types of CBC works is not readily available, this evaluation is based upon a study of grass cutting activities only.
In 2018, CBCs are programmed to deliver works across 6 provinces and 10 islands (Tanna, Malekula, Ambrym, Paama, Efate, Ambae, Santo, Vanua Lava, Mota Lava, Gaua).

In Santo and Efate, CBC works are implemented on major arterial sealed roads.

On other islands, CBC works are, in general, implemented on arterial gravel and earth roads.

4.2 Technical Assessment of the Appropriateness of CBC Works This section focuses on different quality aspects of the grass cutting works undertaken by CBCs.
It aims to assess the adequacy of such works (i.e.

whether they are necessary and undertaken in appropriate locations and therefore contribute to the preservation of the road asset).

It also tries to examine the nature of any requirements for the rectification of works.

The analysis which follows is based around a series of questions.

Do grass cutting activities contribute to the preservation of the road asset, by being implemented on adequate locations (usually ‘engineered’ road sections)?
For effective maintenance, it is essential that rainwater which falls onto the road carriageway is dispersed to the recipient side drain, existing water course or onto sloping ground which takes the water away from the carriageway, keeping the road surface and pavement layer as dry as possible.
This is achieved by regular routine maintenance of the road to ensure there is sufficient camber or 22 CBC Evaluation Report – March 2018 superelevation to effectively shed water from the road surface and to ensure drainage paths are not blocked by overgrown grass or other vegetation.
Grass cutting is therefore an important part of PWD’s preventative maintenance strategy, primarily aimed to ensure that vegetation growth on road shoulders, verges and in drainage ditches does not cause the obstruction of water, which in turn would cause premature deterioration of the pavement, either directly (via erosion) or in combination with traffic (via softening of the gravel pavement layers).
Figure 1 : Obstruction of Water Flow by Overgrown Grass Another reason for grass-cutting is to maintenance sight-distances for drivers, important from the aspect of road safety.

This is however of more concern on those roads where driving speeds and traffic volumes are generally greater, usually sealed roads.

It is, therefore, reasonable to undertake grass cutting only on so-called “engineered” roads for reasons of pavement preservation or road safety.
Roads which require routine maintenance are therefore characterised by the following: (i) camber/superelevation of the road surface with sufficient gradient (2%-5%); (ii) side and cross drainage or other means of allowing water to run away from the road, and (iii) gravel, scoria or sealed/concrete surface.
Grass cutting works undertaken on earth roads, even though the camber and V-drains can sometimes be formed by grader, are generally ineffective as the earth surface will absorb water, become saturated and experience deformation, rutting and potholes whether or not the drainage is working effectively.

However, one exception to this is earth roads where scoria is the predominant in-situ material (as on Ambae).
This section, therefore, aims to establish the proportion of road lengths maintained by CBCs where grass cutting works are contributing positively to the preservation of the asset.

Data supplied by the divisions in answer to the above question reveals that out of a total of 558 kms of routine maintenance planned for 2018, around a quarter is not engineered.
23 CBC Evaluation Report – March 2018 142.05, 25% 416.16, 75% Graph 18: Lengths of Engineered and Non-Engineered Roads Subject to Routine Maintenance Engineered Non Engineered If we look at the same data by province in the two graphs below, we see that the proportions of engineered to non-engineered roads being subject to routine maintenance varies significantly between provinces, with Torba having the highest proportion of the non-engineered maintained road network, followed by Penama, Malampa and Tafea.
The lowest proportion of non-engineered maintained road network is in Sanma and Shefa, where community involvement is predominantly limited to work on the engineered sealed roads.
180.00 160.00 140.00 120.00 100.00 80.00 60.00 40.00 20.00 - 67.00 89.90 34.30 30.85 3.20 - 127.50 114.25 12.75 41.66 Malampa Penama Sanma Shefa Tafea Engineered Non Engineered 24.80 12.00 Torba Graph 19 : Lengths of Road Network subject to Routine Maintenance split into Engineered and Non- engineered by Province 24 CBC Evaluation Report – March 2018 Graph 20 Proportion of the Non-engineered, but Routinely Maintained Road Network by Province We can surmise that the reasons for the engagement of communities on non-engineered road sections may be due to: (i) poor technical knowledge and understanding of the divisional staff; (ii) the increased pressure from communities and/or politicians to involve communities in road maintenance, and (iii) a desire to increase social benefit and sense of equality amongst communities.
We can conclude that a significant proportion of the routine maintenance being undertaken throughout Vanuatu is probably of little benefit and is not contributing to the preservation of the road assets, while utilising a significant proportion of available budget4.

Therefore, the general recommendation is to stop funding regular grass cutting works on non- engineered roads.
It is accepted, however, that occasional vegetation cutting may be required in some areas to keep the road open and allow sufficient sight distance, but these works would be undertaken at a much reduced frequency (probably less than once a year) and only in sections where it is deemed necessary from inspection.

Adopting this strategy would result in significant cost savings that could be usefully used more productively for the benefit of road users.
Does grass cutting on engineered roads always lead to effective water discharge from the road carriageway?

Grass cutting is effective only if it results in water being discharged from the carriageway, usually over the shoulder to the side drain or lower surrounding ground.
In many observed cases where grass- cutting is undertaken on engineered roads the discharge of water from the road is unsatisfactory due to: grass not being cut sufficiently to a uniform height; grass re-growing rapidly before the next cycle of works is implemented, or the road shoulder having lost its initial shape so not providing a positive gradient away from the carriageway, often caused by build up of waste from many cycles of grass- cutting.
In this case, grass cutting itself does not represent an adequate solution.

Additionally, shoulders should be re-graded to the required gradient with a slight fall away from the road.

The test for each division is to simply visit roads during or after heavy rains to identify whether water falling on the road is being quickly discharged.
If water is ponding on or along the edge of the carriageway, some remedial action 4 See later section for cost-effectiveness analysis also 25 CBC Evaluation Report – March 2018 to the shoulders is probably required.
Where very fast growth of grass between cycles results in ponding of water on the road carriageway, alternative solutions to vegetation control could be considered, such as re-planting with a different type of grass, the introduction of different grass cutting technology that can ensure uniformity and quality, or use of glyphosates or thermal technology to control the vegetation growth (or a combination of these).
There have been several instances within the last two years when the grass on the Efate Ring Road has been cut before an inspection, but during or after heavy rain water can still be seen on or immediately alongside the road carriageway.

This phenomenon often occurs in areas where there are issues with shoulder gradients, uneven grass cutting and grass/vegetation cuttings left at the roadside obstructing water flow.
These examples are presented anecdotally only, but they give sufficient indication that the current approach of simply cutting grass without giving due consideration to the impact on water run-off is often ineffective.

PWD should make a visual assessment of each road during or after heavy rain and adjust their approach to routine maintenance where necessary to ensure water is removed completely from the road within a reasonably short time without ponding.
Mitigation measures to improve water run-off may include:  reinstatement of shoulder gradients to 4 - 8%;  improved and more uniform grass-cutting by using equipment - using a tractor sidearm slasher produces better results than hand-held trimmers, whilst trimmers produce better results than bush knives;  considering the use of vegetation control technologies such as glyphosates or thermal technologies, and  considering the use of performance-based maintenance contracts which incentivise contractors to avoid ponding of water.
4.3 Quality Comparison of Different Grass-Cutting Methods Grass cutting works implemented by CBCs often utilise bush knives as a low-cost solution which is readily available to rural communities.

However, during 2017, some communities implemented the works by using hand-held trimmers (string-cutters) and some even sub-contracted their grass-cutting works to a tractor-owner.
The adoption of these three different methods during 2017 has allowed us to undertake a comparison of outputs in terms of quality and efficiency.

Quality comparisons were subjective based on the view of various divisional staff and R4D’s Road Operations Specialists (ROSs).

The scoring focused on the level of uniformity and height of the grass cutting achieved when works were implemented using the different tools.

Not surprisingly, the work undertaken by the tractor slasher achieved the best quality.
The assessors regarded the quality achieved to be 88% better than that achieved with bush knives.

Grass cutting by trimmers was considered to produce quality 80% better than works undertaken by bush knives.

It is important to note that the work undertaken by the tractor slasher was in fact a mix of tractor slasher and trimmers, as trimmers were used after the tractor to reach all the tight inaccessible areas missed by the tractor (e.g.

around drains, traffic signs, etc.).


Graph 21 presents the quality assessment findings.

26 CBC Evaluation Report – March 2018 100% 80% 60% 40% 20% 0% Bush knives Trimmers Tractor slasher Graph 21: % Quality Improvement in Grass-Cutting Compared to Bush-Knives The tractor slasher with associated trimmers is the preferable option from a quality perspective.

In general, and especially when undertaking works using the tractor slasher, grass is cut in a uniform manner and to the appropriate height.
However, the application of a tractor with slasher is limited to those roads with well-formed, uniform shoulders/verges/V-drains such as East Coast Road, Santo.

Also, the tractor must be supplemented with a trimmer to achieve full coverage.

Trimmers are seen to provide almost the same level of quality as a tractor-slasher, and have the added advantage that they can be used on any road.

This is discussed further later in this report.


Summary of Recommendations on Quality  Grass-cutting on non-engineered roads should be avoided, and replaced with occasional vegetation trimming to keep roads open only when required.

 Grass cutting must result in water being discharged from the carriageway effectively.

Where inspections show that this is not happening, further mitigation works should be undertaken such as shoulders re-shaping or removal of roadside debris.
In some situations, performance-based contracts with a requirement to avoid ponding on the carriageway could be consideed.

From a quality perspective tractor slashers and trimmers should be the preferred equipment for the grass cutting works, but trimmers only provide a very good result also.
  4.4 Comparison of Cost-Effectiveness This section aims to: (i) assess how the CBC model compares to other modes of delivery in terms of cost- effectiveness; (ii) identify potential areas for the improvement of the CBC model, and (iii) identify potential areas for the application of other contracting models.

Various sources of information were used to extract or derive the following information:  Workplan 2018 CBC projects information: no.
of contracts, unit rates, scope of works, quantities  Divisional staff and ROS(N) inputs: lengths of engineered/non-engineered road sections, types of pavement  Tractor-owner subcontractor to CBCs: actual quantities of works undertaken, payments received, productivity rates  General R4D and PWD experience: productivity rates, payment terms and performance, contractor’s overhead and profit, PCPO and SCPO time inputs  Available PWD records: salaries of management and supervision staff, equipment costs 27 CBC Evaluation Report – March 2018 For this exercise, each type of equipment considered (bush-knives, trimmers and tractors) was assumed to be used by its most likely contracting form as follows: (i) (ii) (iii) bush knives used under CBC contracts trimmers used under IBC/small scale contractor contracts tractor-slasher supplemented with trimmer used under a larger national contractor (NC) contract.
The costs of grass-cutting by CBC’s using bush-knives were calculated on a unit basis (for each m2 of output) by considering the following costs: (i) labour and other community costs; (ii) PCPO and SCPO oversight and management costs; (iii) administrative costs (procurement and payment processing), and (iv) cost of equipment purchased for a typical community contract.
The costs of grass-cutting using trimmers by IBCs were derived from: (i) calculated unit rate (inclusive of cost of equipment, personnel, overheads and profit); (ii) VAT; and (iii) administrative costs.

In the case of National Contractors using tractors, costs were derived from: (i) unit rates used when CBCs sub-contracted a tractor-owner; (ii) VAT, and (iii) administrative costs.
The cost of other PWD staff such as Divisional Manager, Divisional Engineer and Site Inspector were considered to be similar across all equipment types and were therefore ignored for the purpose of this comparison.

Typical productivity rates were derived from PWD and R4D experience across Vanuatu.

The graph below compares the productivity of a single worker with a bush knife (150m2/day) against a single worker with a trimmer (2,000 m2/day) and a single tractor with slasher (28,000 m2/day).
30000 25000 20000 15000 10000 5000 0 Bush knives Trimmers Tractor slasher Graph 22: Grass-Cutting Productivity Rates Even if we consider the output of a typical eight-man gang, productivity increases to only 1200 m2/day, still far less than the other types of equipment with a single user.

Productivity rates are the main factor influencing the unit price of grass cutting works under the three considered options, and have been used to derive unit costs expressed as vatu/m2 which are shown in the graph below.
28 CBC Evaluation Report – March 2018 12 10 8 6 4 2 0 CBC IBC NC Unit rate (VUV/m2) VAT (15%) Equipment cost PCPO Time Cost SCPO Time Cost Administrative costs (VUV/m2) Graph 23: Derived Unit Costs of Grass-Cutting The table below shows the cost data used to derive the graph above.
Table 2: Detailed Cost Information for Different Grass-Cutting Equipment Types As can be seen from the above, the delivery of grass-cutting works when using CBCs with bush-knives is around four times more expensive than other methods.

This suggests that significant savings could be achieved by using more efficient equipment for grass cutting activities.
If we combine this potential cost saving with the earlier finding that much grass-cutting is undertaken on non-engineered roads when it is unnecessary, we are able to generate even further potential savings.

Graph 24 below shows the value of 2018 routine maintenance split between useful work on engineered roads and less useful work on non-engineered roads.
29 Type of equipment requiredBush knives (CBC)Trimmers (IBC)Tractor slasher (NC)TOTAL Grass cutting unit costs (VUV/m2)10.57 2.30 2.89 Productivity rate applied (m2/day)150 2,000 28,000 Unit rate (VUV/m2)92.02.5VAT (15%)00.3000.375Equipment cost 0.1800PCPO Time Cost1.0700SCPO Time Cost0.06 00Administrative costs (VUV/m2)0.260.0050.02 CBC Evaluation Report – March 2018 15,861,888 , 21% 58,958,280 , 79% Engineered Non Engineered Graph 24: Value of Useful Routine Maintenance on Engineered Roads vs Less useful Routine Maintenance on Non-Engineered Roads The value of grass-cutting work to be undertaken on non-engineered roads in 2018 amounts to around 21% of the total grass-cutting costs across Vanuatu (almost 16m vatu).
The graph below suggests that if we reduce work on non-engineered roads and we undertake grass-cutting by tractor, we could generate significant savings.
80,000,000 70,000,000 60,000,000 50,000,000 40,000,000 30,000,000 20,000,000 10,000,000 - Bituminous seal Concrete Gravel Earth TOTAL Total current cost Total theoretical future costs Graph 25: Potential Savings on Grass Cutting Works (VUV) Total savings across the whole network based on the two assumptions applied would amount to around 51m vatu (equivalent to 68% of the total 2018 costs for grass-cutting).
These savings could utilised to expand routine maintenance activities to a greater proportion of the engineered road network, which would be of enormous benefit to Vanuatu.

Graph 26 below shows the number of kilometres of engineered road that could be tackled by each of the three different equipment types based upon the 2018 budget available.
This shows that by switching equipment (and contract) type, we could maintain a much greater length of network than is currently achieved using the CBC/bush-knife model.
30 CBC Evaluation Report – March 2018 2,500.00 2,000.00 1,500.00 1,000.00 500.00 - CBC IBC NC Sealed/concrete Earth/gravel Graph 26: Lengths of Engineered Network that can be Maintained using Different Types of Equipment Summary of Recommendations on Cost Effectiveness  Grass-cutting on non-engineered roads sections should be avoided (repeated from quality recommendations)  Grass-cutting using tractors fitted with slashers is strongly recommended to be trialled on those engineered roads with uniform shoulders and verges, which are usually the sealed/concrete roads on the network  Grass cutting on engineered gravel roads should be undertaken by small scale contractors, such as the IBC’s or similar, using trimmers.
 CBCs should be re-directed to undertake maintenance works primarily on earth roads, including earth road pothole patching and grass cutting for aesthetics, encroachment and sightlines.

4.5 Other Considerations This section deals with other technical and administrative aspects of the CBC model outside quality and cost effectiveness considerations.

A number of key areas are explored and analysed and key conclusions drawn.
4.5.1 Procurement and Management Arrangements The procurement of community-based contracts is currently implemented through direct agreements between the PWD and the community contractor.

Unit rates for works are defined by PWD and accepted by the communities as being fair.

Such arrangements have significant practical advantages over any form of competitive bidding, as significant time and effort is saved during the pre-contract preparation stage.
However, the model is not in line with the Government’s procurement regulations, which does not allow direct award.

In the case of IBCs, an exception to allow direct awards has been issued by the Government, but this is not in place for CBCs.

Management arrangements for CBCs are complex because of the involvement of both divisional and PWD HQ teams, and the necessary liaison with the Provincial Government and local ‘kastom’ structures and communities.
Now that CBCs have been in operation for some time, it would be useful to undertake a review of CBC management arrangements with the objective of creating an updated set of roles and responsibilities for all those involved in CBC management and administration.

31 CBC Evaluation Report – March 2018 This report’s qualitative assessment of CBCs gained through discussions with communities, suggested that there are issues with the quality of works supervision.
Analysis of the CBC contract information for 2017 supports this claim.

In 2017, there were no variation orders issued for community contracts, and there were also no requests for the rectification of works issued.

This can mean only one of two things: (i) either both design and the implementation of works for all 198 contract was flawless, or (ii) payments are made generically by simply following the BoQ quantities without the necessary inspections and measurements taking place.
The following recommendations are made in relation to procurement and management of CBC contracts: Summary of Recommendations on Procurement and Management of CBCs   PWD should apply for necessary exclusions from GoV to allow direct award of works to communities PWD should review, simplify and re-issue its roles and responsibilities for those involved in the implementation of CBC contracts 4.5.2 Community Performance and Relationships The power that PWD can exercise over a community under the current contractual arrangements in the event of poor performance is limited.
As an example, in 2016, a Tanna community that was supposed to be implementing maintenance works from Imanaka to Lowiaru proved difficult to manage.

The works that they implemented were unsatisfactory.

However they did not allow PWD to appoint any other contractor to do the work as they had developed a sense of entitlement to the contracts and a sense of ownership over their road section.
What resulted was a tense relationship with little progress made to negotiate a settlement whilst the road suffered severe degradation as it was not maintained for over a year.

Such examples, although rare, show that the model is prone to failure in the event of a serious dispute.

Under a conventional contracting model this scenario would not occur as a contractor is not able to claim ‘ownership’ of a road section and prevent others from working there.
On this basis, the following recommendations are made: Summary of Recommendations on Community Performance   PWD should ensure communities understand their commitment to performance under a CBC contract.

PWD should consider including measures in the standard CBC contract to deal with poor performance and should include a mechanism to refer disputes to an independent body for faster resolution.
4.5.3 Health and Safety of CBC Workers and Road Users According to the last available Divisional Monthly Reports for 2017, the majority of inspections undertaken to evaluate CBC’s compliance with Workplace Health & Safety (WHS) requirements was regarded as being “partially” compliant, when observed across all of the provinces.

This could mean that some workers on the site have been wearing protective equipment, while some have not, or some communities have erected the traffic cones, while others have not.
There are several safety issues related to the CBCs:  it is difficult to ensure a high level of compliance due to the high number and frequent rotation of workers engaged by communities in the works.

Improving compliance would require 32 CBC Evaluation Report – March 2018 intensive monitoring, a system of penalties and incentives, and frequent re-training of new workers.
 works for one cycle of grass-cutting typically only last for one or two days, making it difficult to supervise and inspect WHS compliance.

 CBC’s are often working on high speed roads with only traffic cones to provide protection to workers by controlling and slowing traffic, thus exposing workers to danger from passing vehicles.
While the safety risks are probably negligent for the low-trafficked and low-speed roads, they should be an important consideration when deciding on the appropriateness of the CBC model for undertaking the grass cutting works on higher-speed arterial, urban or semi-urban roads.
In light of the above, the following recommendations are suggested: Summary of Recommendations on Health and Safety  To improve safety, PWD should consider not using communities to cut grass on high speed arterial, urban or semi-urban roads.

 A health and safety risk assessment should be undertaken for each individual CBC contract identifying potential hazards and defining mitigation particular to each site.
4.5.4 Increasing CBC Scope of Works This section considers the ability or potential of CBCs to expand their scope of works beyond simple grass-cutting and vegetation control.

Expanded scopes could include works such as the removal of debris from the blocked drainage during rainy periods, production of gravel, surface repairs and pothole patching.

These are considered below.


Removal of Debris from the Carriageway, Blocked Drains and Drainage Structures Approximately 50% of communities have already had experience in undertaking these works in the past.

Removal of debris is undertaken in cycles, usually before and after the rainy season.

The quality of clearance works by CBCs has usually been very satisfactory.

Communities have proven themselves to be successful at implementing these works due to their simplicity and proximity.
Consideration should be given to strengthening this important CBC role.

Quarry Extraction Communities from the Banks Islands in Torba have already proven their ability to undertake these works during 2017.

The production of gravel of was undertaken on Vanua Lava, Gaua and Mota Lava.

On Vanua Lava and Mota Lava, community members were employed and supervised by IBCs.

On Gaua the community was contracted directly.


All communities used exclusively labour-intensive methods of work for this activity but achieved good production rates and fulfilled their contract obligations.

Our experiences from these locations tell us that communities can be solely contracted for these types of work in Torba.

However, extending this scope to communities in other provinces is untested.
The advantages of employing communities directly, in this particular case, were:   the works were undertaken intensively over a prolonged period of time, therefore providing stable employment and a significant and welcome source of income for the community members communities are often better placed than independent contractors to deal with land issues It should be possible to organise gravel extraction work such that it provides continuous employment to a small permanently employed team making best use of available tools which are also used 33 continuously.
This should reduce overall costs and with appropriate training and supervision result in a skilled, productive team.

CBC Evaluation Report – March 2018 Surface Repairs on Earth Roads Maintaining an earth road surface by filling depressions using locally available soils is a simple task which could easily be undertaken by communities using basic equipment.

This could be an important function on some important feeder roads which serve communities, and should be explored further by PWD.
Pothole Patching on Gravel Roads Pothole patching of gravel roads is currently being trialled during 2018 by communities on Santo.

The results of this trial will tell us whether communities are able to successfully undertake these works and give us the ability to assess quality of works, cost and resilience of the completed works.

Pothole patching on gravel roads requires some training to ensure patches are square with equal depth of excavation.
Infill gravel should be wetted to be close to its optimum content to allow easy and dense compaction.

Pothole patching requires the use of some specialist tools: hand-tamper for compaction, pick-axes and wheelbarrows.

These have been provided by PWD in the Santo trial.

The gravel material for pothole patching has been provided by PWD and placed at regular intervals along the road.

This is therefore a collaborative effort between PWD and the communities, and the results will be therefore be interesting.
34 CBC Evaluation Report – March 2018 5 CONCLUSIONS AND RECOMMENDATIONS The CBC model has provided PWD with a unique opportunity to engage directly with communities.

The theoretical underpinnings of the structure are sound in that it provides opportunities to engage communities to play a role in contributing to the maintenance of the rural road network in their area and, as a bonus, these communities earn additional income to improve community facilities.
A secondary benefit is the support provided to maintain and enhance existing community decision- making systems and structures.

From a technical perspective the analysis has revealed a number of quality issues and cost effectiveness issues between traditional “bush-knives” towards more high level equipment such as tractor-drawn slashers and trimmers.

The technical analysis also raised some other issues which require consideration.
The general perception amongst communities is that the CBC model is useful and popular.

However, it is evident that some approaches need to be modified to further improve the model.

A number of recommendations are raised below for PWD consideration.

Issue or Area for improvement Recommended Actions 1.

Safeguards (Safety, Social, Environmental) 1a In some instances, community- based workers are being exposed to work alongside live traffic without safety training and without PWD- issued safety equipment.
 No workers should be undertaking works prior to receiving safety equipment and training on how to use it.

This should be written into a revised CBC contract.

 Safety training sessions (with simple handout booklets) should be held at the beginning of each year for groups of communities.

Attendance should be compulsory for all workers.

Consideration should be given to issuing a PWD ‘permit to work/safety licence’ to each attendee, without which he/she cannot work.
 PWD should work with each community to undertake a health and safety risk assessment for each particular work site prior to commencement of works.

This may identify special hazards, such as blind corners, working next to water, etc.

which require non-standard mitigation such as ‘slow down’ signs or warning flags.

PWD should take contractual responsibility for providing additional safety equipment if the risk assessment deems this necessary.
1b The perceived pressure to employ more women on physical works is causing security and safety concerns for some female workers who are having to walk home from their work sites long distances in the dark.
There are also complaints  Review, and if necessary, amend training materials and associated guidance documents to make it clear that promoting gender equality means making equal work opportunities available for both men and women, but places no compulsion upon women to work if this is not practical in particular community or family settings.
35 Issue or Area for improvement Recommended Actions CBC Evaluation Report – March 2018 from some women that combining home duties with physical roadworks is causing them overwork and stress.

The direct award of CBC contracts to communities without competition is not allowed under current Vanuatu laws and regulations.

No exemption has been issued as is the case for IBC direct awards.

1c 2.
1c 2.

Contracts and Contract Administration 2a 2b 2c The transition from the old-style CBC contract signed by an individual to the newer form of CBC contract which is signed on behalf of the community by a community leader is not fully understood by most CBCs.

Frequent late payments to communities, with pressure being placed on them to start new cycles of grass-cutting before being paid for the previous cycle.
The current form of routine maintenance contracts which define inputs to be undertaken by the community or contractor are not necessarily producing the required outcomes.

For instance, after grass-cutting there are many examples of standing water remaining on carriageways after rain causing pavement deterioration.
 PWD should apply for an exemption allowing the direct award of CBC contracts based upon standard market rates  Develop and deliver a standard training course to all communities on contractual responsibilities and management arrangements.

At least two from each community should attend such training with attendance recorded.

 Make a review of the payment process and responsibilities for each step in detail.

Take appropriate measures to improve and to meet contractual 30-day terms.
If this cannot practically be achieved, consider changing the contract terms so at least communities understand when payments are to be received.

 Divisional management and engineering teams should assess road sections in more detail to determine if grass-cutting alone will improve drainage and therefore extend pavement life.

Where additional works such as shoulder and verge re-shaping to improve water run-off are required these should be implemented.
 PWD should consider the urgent trial of a simple performance-based longer term routine maintenance contract with payment linked to keeping vegetation below a certain height and avoiding standing water on carriageways after rain by reshaping shoulders and clearing drains where necessary.

2d Some communities stated that they were unaware of grievance procedures and complained that their questions and complaints were often unheard.
Conversely, PWD have had difficulties dealing  PWD to consider adjust their management structure and where necessary amend the wording of CBC contracts to strengthen two-way grievance/complaints/sanctions procedures.

A process for escalation of unresolved issues to an independent entity should be included in CBC 36 Issue or Area for improvement Recommended Actions CBC Evaluation Report – March 2018 with under-performing communities.

3.
3.

Management Structure 3a CPO performance is questionable, with reports of lack of engagement by communities, lack of supervision of works, very little contract administration and lack of training being provided to communities.

4.

Capacity Building/Training 4a Many communities reported that they had not received training and through discussion a lack of knowledge of important topics has been demonstrated contracts to give both parties added comfort and protection.
 As part of the wider HR reform strategy being developed in PWD, undertake a thorough review of the role and responsibilities of CPOs and other team members including a review of the practicality of performing prescribed duties across multiple contracts on different islands and the split of responsibilities for managing CBC contracts within each divisional team.

Contract administration, technical supervision and payment certification all require considerable strengthening.
Revised organisation structures, position titles, reporting lines and job descriptions should be issued.

It is likely that a major restructuring of divisional teams is required.

 Once this revised management structure has been developed and agreed, provide mandatory training to all staff involved on their revised roles and responsibilities in relation to CBCs and wider maintenance activities.
 A different approach to the provision of training is recommended, with PWD HO taking more responsibility for: (i) the development of standard training modules and documentation, including videos or presentations which can be shown on tablets in the field when necessary, (ii) annual community training roadshows in January/February to cover all islands where CBC contracts are being used, (iii) improved attendance records and issuance of ‘permits to work’ for workers when training is completed.
Training modules should cover as a minimum: technical basics and specifications, workplace health & safety and risk assessments, social and environmental safeguards, contract provisions and management responsibilities.

Contracts should be amended to include mandatory training for workers.

5.
5.

Design and Preparation of Routine Maintenance Contracts 5a Many CBC contracts are being implemented which include multiple annual grass-cutting cycles on non-engineered roads where unblocking of drainage paths to preserve pavement life is  Divisional management and engineering teams should pay much more attention to the detail of CBC contracts when setting them up to ensure that scarce funds are not being wasted on unnecessary works.
However, it is accepted that even on non- engineered roads, some grass-cutting and 37 Issue or Area for improvement Recommended Actions CBC Evaluation Report – March 2018 unnecessary.

Funds are being wasted on these works.


5b Some CBCs have reported that the first grass cut each year is very difficult and should therefore attract higher rates then subsequent easier cuts 5c Grass cutting undertaken by CBCs using bush-knives is substantially more expensive than if undertaken by small-scale contractors using trimmers or larger contractors using tractor/slashers.

Very large cost savings can be made by switching grass-cutting to more economic methods.
5d Communities dissatisfied with length and scope of works and want the opportunity to do more vegetation control may be necessary to prevent excessive encroachment of vegetation into the road, to keep paths open for pedestrians, to ensure safe sightlines are maintained and for aesthetic reasons especially in areas frequented by tourists.
 Divisional management and engineering teams should be given the flexibility to apply a surcharge to standard grass-cutting rates when works are more difficult than typical works.

This should be assessed on a case-by-case basis as it will not apply to all sites.

Guidance for divisional teams should be developed to make this flexibility clear.
 Although this is a transition that will require careful handling, PWD should implement a strategy of using contractors to undertake routine maintenance on urban, semi-urban, sealed or highly trafficked roads.

This transition will free up additional budget to allow expansion of routine maintenance activities across more of the road network  CBCs should be used to undertake routine maintenance works only on less trafficked local feeder roads which are likely to be earth or sometimes gravel.
 Guidance for divisional teams should be developed to make the choices clear.

 Divisional management teams should consider opportunities to increase the scope of work undertaken by communities where it makes economic and practical sense to include them in debris clearance, earth road surface repairs, gravel road pothole patching and quarry extraction.

This evaluation has come as a wake-up call for the CBC program, as it is clear there are many serious issues which require urgent attention.
Many of the recommendations listed above require a fundamental restructuring of the approach to routine maintenance by PWD.

These issues were unapparent from a monthly review of CBC contract data in PWD Head Office, so this evaluation has proved to be a very useful exercise.

Given the large amount of issues and recommendations contained in this report, it is further recommended that R4D develop a CBC Guidance Note to be adopted by PWD before the end of the current phase of R4D on 30 June 2018.
It may be necessary to hold a workshop with PWD to finalise the CBC Guidance Note.

The R4D team would like to thank both DFAT and PWD for their guidance and support in producing this evaluation report.

38 ANNEX 1 CBC SURVEY TOOL – QUANTITATIVE SURVEY CBC Evaluation Report – March 2018 Background Information Name of the community Name of community leader Location (Province, island) Contract length (km of road) Initial Questions 1.

Does this community have a current CBC contract 2.


How did you/community hear about the CBC concept 2.

Yes 3.

No 4.

Don’t Know 1.

Community Leader 2.

PWD 3.

Provincial/District Government 4.

Neighbouring Community 5.

Other CBC Training Program 3.

Has the community received road maintenance training (3 modules) through PWD?

1.

Yes (go to question 4), 2.

No.

3.

Don’t know (go to question 7) 4.


In thinking about this training, in your opinion how 5 4 3 2 1 (5 excellent – 1 poor) effective was it?

5.

Have you and the community applied the knowledge Yes No Don’t Know form the training in completing works?

6.

What aspects have you/community applied?

1.

Grass-cutting, 2.

Tree Pruning 3.

Drain Clearing 7.

What additional training do you think you need more Drop down menu of?

CBC Funding Mechanisms 8.


Are you aware that funding has been made available to 1.

Yes (go to question 9), 2.

No, 3 Don’t the community for CBC works?

Know (go to question 14) 9. Who collects and handles the funds 1.

Chief 10.

Where has that money been spent?

1.

New church, 2.

Community resource etc.

11.

How happy are you with the distribution of 5 4 3 2 1 (Very Happy – Not Happy at all) money in the community 12.
To what extent has the provision of money 5 4 3 2 1 (A lot – We are worse off) increased the well-being of the community 13.

In thinking about well-being, what could be done Drop Down Menu to improve or streamline payments?

Community Engagement 39 CBC Evaluation Report – March 2018 14.

Has the CBC model promoted a greater sense of community and community engagement Yes (go to question 15) No Don’t Know (go to question 16) 15.

If yes, what is the community doing differently Drop down menu than before 16.
Has the CBC model facilitated broader engagement with groups who may not have had the chance to participate (youth, disabled and women) Yes ( go to question 17) Know Don’t Know (go to question 18) 17.

18.

If yes, please provide an example.

Short paragraph - qualitative To what extent does the community believe it is 5 4 3 2 1 (very much – not at all) the “owner” of the road?

Grievances 19.

Has the community experienced any grievances 1.

Yes (go to question 20) 2.

No.

3.


Don’t with regards to the CBC contract know (go to question 23) 20.

What was the cause of the grievance?

Drop Down Menu 21.

Was it resolved?

1.

Yes 2.

No, 3.

Don’t Know.

22.

If not, why wasn’t the issue resolved?

Short answer 23.

IS the community aware there is a formal grievance mechanism through the CBC?

Yes (go to question 24) No Don’t Know (go to question 27) 24.

Was the mechanism used in resolving the community grievance(s)?
Yes (go to question 25) No Don’t Know (go to question 26) 25.

How effective was the CBC grievance mechanism 5 4 3 2 1 (5 excellent – 1 poor) in resolving the conflict?

26.

What could be done to improve the mechanism?

Drop Down Menu Concluding Comments 27.

What is your overall impression of the CBC 5 4 3 2 1 (5 excellent – 1 poor) contract system?

28.

Any additional comments you would like to add?

– qualitative short response.

40