Source document: https://docs-lawep.s3.us-east-2.amazonaws.com/1697635381524.pdf (Australia)
Scrhilc inquiry mto the adequacy of cx1st,ng offence~ in the Commonwen\th Cr1mmill Code 11nd of state and lcrntory crtmmal liJWs to c.aptute cyberbuHying The Alannah & Madeline Foundation's Response Summary Legal frameworks and criminal law are only one measure through which government can support our community to effectively reduce and manage the negative impact of cyberbullying on individuals, families and the community.
This is especially the case regarding children and young people in relation to the Commonwealth Criminal Code and existing state and territory criminal laws.

The Alannah & Madeline Foundation (the Foundation) continues to advocate for laws that are capable of managing issues of serious or grievous bullying, but which are nuanced enough to take into account the context and reality of children and young people's lives.
The Foundation's commitment and ongoing support for adequate systems that help the community to understand, identify, address and reduce the impacts and incidence of cyberbullying include: • The National Centre Against Bullying In 2002, the Foundation recognised bullying as a major form of violence experienced by children, and set up the National Centre Against Bullying (NCAB).
NCAB is a peak body working to advise and inform the Australian community on the issue of childhood bullying and the creation of safe schools and communities.

including the issue of cybersafety.

In 2013, NCAB, via the Foundation, convened a 'Bullying.

Young People and the Law Symposium· which brought together more than 100 legal and law enforcement experts.
• eSmart Schools and eSmart Libraries Established in 2010, this whole-of-school approach is the most effective way to equip a community w ith the skills, knowledge and behaviours to reduce rates of cyberbullying and improve responses to cyberbullying by children, young people and the community as a w hole.

Currently in over 2,300 schools nationwide, over 90% of participants agreed that eSmart assists schools to embed smart.

safe.
safe.

responsible behaviour across the curriculum.1 • Digital licence The Digital Licence is an online education tool.

which uses quizzes, videos and games to teach school age children about how to be capable, confident and safe users of the internet and digital technologies.

The adequacy of existing laws relies on ongoing programs and systems such as these.
These programs ensure the community is educated and aware of the laws related to cyberbullying, but also have the skills and knowledge to effectively manage themselves and their behaviours in digital environments.

The Alannah & Madeline Foundation would therefore recommend to the inquiry: 1.

The need for a national, uniform legal definition of what constitutes bullying.

The Foundation (via NCAB) believes there is a need to properly define bullying - including cyberbullying as a specific type of bullying.
Bullying has been defined by the Commonwealth in regards to the workplace 1 2015, eSmart Schools Independent Evaluation.

Summary PDF: www.esmartschoofs.org.au/ Documents/ eSS Evafuation.Pdf Alannah & Madeline Foundation 2 setting - we advise the Senate uses this to arrive at an accepted national definition with school aged children in mind as well.

2.

Granting the Commonwealth power to legislate on bullying as a crime (beyond its current jurisdictions of workplaces and some electronic contexts).
However, this needs to avoid the unnecessary criminalisation of children and young people under 18 in relation to their use and experience of social media and other emerging technology platforms.

The recent 'sexting' laws wrongly stigmatised some young people as peddlers in child pornography and gave them a criminal record as a result.

3.
3.

Increased formal support for early intervention, education and related programs for children and young people that improve their understanding of cyberbullying, their legal responsibilities and citizens (including digital citizens) and supports them to develop values and behaviours that lead to respectful behaviour both online and offline.

eSmart is now in over 2,300 schools across Australia despite not being mandated at a Federal level.
We advocate for eSmart to be an integral part of Australia's education system in relation to cyberbullying and cybersafety, and call for its national implementation and support.

There is a growing gap in who gets access to world class cybersafety education, and we are concerned that while eSmart is increasingly the program of choice by schools in higher SES communities, all students would benefit from this program.
About the Alannah & Madeline Foundation The Alannah & Madeline Foundation is a national charity established in memory of Alannah and Madeline Mikac, aged six and three who were tragically killed with their mother and 32 others at Port Arthur, Tasmania on 281 h April 1996.

The Foundation believes that no child should feel unsafe and that confident, secure, resilient and inspired children are our hope for the future.
The Foundation works across the spectrum of prevention and care, identifying issues that adversely affect the wellbeing of children and work to recover and protect them from bullying and violence through state-of-the-art, evidence-based programs and products.

The Foundation works mostly with children aged 16 and under in their homes.

schools and in the community to care for those who have experienced or witnessed serious violence; reduce the incidence of bullying, cyberbullying and other cyber risks.
The work is undertaken collaboratively w ith the best minds to identify and reduce significant threats to children's safety through innovation and partnerships.

The Foundation speaks out on the need for systemic change to build a supportive and safe society.

To date, the Foundation has helped over 2.1 million children through its work.

The Foundation's work is underpinned by a set of guiding principles: • Child centred • Innovation driven • Evidence based • Values led • Ethical partners.
The Foundation commenced its work in the area of cybersafety almost ten years ago when it became increasingly apparent that the development of online networks created a new challenge.

Since that time we have become a leader in the space developing innovative programs, and leading the way towards a society that understands and values eSafety.
Alannah & Madeline Foundation 3 Our approach to this inquiry In this submission, the Foundation addresses three key areas that cover all aspects of the inquiry's investigation.

The perspectives and advice are based upon: • • The Foundation's expertise and experience working w ith/ for children and young people on whom these laws have the potential to impact upon in unintended ways.
The Foundation's deep knowledge and success in developing and delivering community programs that support the safety and well-being of children and young people.

We w ill be addressing the inquiry in the following three areas that are reflected in the next three sections of this response.

1.

Children, cyberbullying & the law (Questions A - C} a.
We will address the issues relating to social media services and specifically the application of section 474.17 of the Commonwealth Criminal Code where it relates to children and young people.

2.

Beyond the criminal code (Question D) a.

We will discuss how the Foundation's work will be able to complement new / revised legal frameworks to combat cyberbullying in school communities.
We will focus on the importance and value of this work especially as an early intervention and preventative measure, but also as a vital component in coordinating community education of new laws and the proper procedures that must be enacted at an inst itution level.

3.

Holistic approach (Question E) a.


We will offer insight into the interconnected nature of education, legal and community frameworks and programs, emphasising the value and significance of a holistic approach in combating cyberbullying in Australia.

We see the provision of the Foundation's eSmart Framework to all schools in Australia as a front-line measure to address the emerging 'digital divide' between low and high SES schools, which increases the number of 'youth at risk' online.
Alannah & Madeline Foundation ' 4 Children, cyberbullying & the law Addressing: a. b. c. the broadcasting of assaults and other crimes via social media platforms the application o f section 474.17 of the Commonwealth Criminal Code 'Using a carriage service to menace, harass or cause offence', and the adequacy of the penalty, particularly where the victim of cyberbullying has self-harmed or taken their own life the adequacy of the policies, procedures and practices of social media platforms in preventing and addressing cyberbullying.
The challenge of defining cyberbullying Over recent years, the legal frameworks seeking to address cyberbullying have continued to be interrogated and changed.

However, the law still deals predominantly with singular incidents of bullying behaviour rather than addressing the wider range of bullying behaviours that are likely taking place within the social context of the participants.
This is specifically the case with cyberbullying where the focus is on an aspect of bullying (and not the activity in its entirety) such as in the application of section 474.17.

What this means is, a person may engage in bullying behaviour both offline and online, over a sustained period of time and they may use different online services to do this.
This is more likely to result in a tragic incident of significant self-harm or suicide by the victim - it is often the cumulative effective of what is described in section 474.17, more so than a single (or several) identifiable incidents of 'Using a carriage service to menace, harass o r cause offence·.

We are in the early stages of creating a legal definition of cyberbullying in relation to criminal law.
The rapidly changing nature of technology means that the form cyberbullying takes continues to morph as new ways and means of using technology emerge.

The law only becomes relevant when the bullying behaviour itself constitutes a crime.

This means that any attempt to invoke the law is really an attempt to fit the bullying conduct into a legal framework that has been designed for something else.
A definition Cyberbullying is a form of indirect or covert bullying conducted via digital technologies and is not easily observed or acknowledged by others.

The perpetrator repeatedly communicates messages that are hostile or aggressive and intended to inflict harm on the recipient2• The risk of criminalising children In the case of children, there are added complications in relying on the criminal law because of diminished responsibility between the ages of 10 - 14 and absolute immunity prior to that age.
This is significant because our understanding and experience of cyberbullying as a phenomenon is most pronounced amongst children, specificaUy teenagers.

It is well documented that young people are some of the highest percentage users of social media in our society, and it is a significant part of their social world.

It is also acknowledged that they are more likely to use the latest social media services that offer new forms of online engagement.
This makes it more difficult for the law to maintain pace with technolog ical developments and the types of social interactions that occur on these services.

2 Spears, Keely, Bates and Katz: Research on youth exposure to, and management of, cyberbu llying incidents in Australia: Part A: Literature review on the estimated prevalence of cyberbullying involving Australian minors, p 5.
Atannah & Madeline Foundation s Relatedly, the 'start-up' (innovation) culture of the technology industry prioritises 'testing in the marketplace' and responding to user feedback to improve their services - this takes precedence over 'user safety by design'.

In their fi rst year on the market, social media platforms can often have significant cyberbullying and harassment issues, due to their lack of monitoring and reporting processes.
Young people are often the 'play-testers' in this environment, as their age group has a higher proportion of 'early adopters'.

Any moves to improve and further regulate the 'policies, procedures and practices· of social media platforms would need to address these emerging services and the risks they pose to the young people that trial them in their earliest iterations.
A key point of contention in many cyberbullying incidents is whether an offender is deliberately and repeatedly engaging in hurtful behaviour.

If they are not, it may simply be necessary to learn better online behaviour.

Jf they are, it is important to take it seriously and address it effectively.

For children.

the way they use and understand social media is challenging our own understanding and expectations around everything from privacy to intimate relationships.
This is especially true for children under 18, as their experience of the world has elements of experimentation and exploration.

While cyberbullying should not be tolerated, young people's use and experience of social media platforms may be interpreted by the law in different ways than the intent an d purpose for w h ich the platforms are being used.
Th is has been discussed and acknowledged in issues around sexting and there are aspects of cyberbullying that have similar nuance that make laws like section 474.17 limited in their capacity to deal with the way children use and explore the t echnology.

As mentioned in the above section on 'The challenge of defining cyberbullying', we advocate that any new or altered law acknowledges the range of behaviours occurring in any given bullying situation.
A valuable precedent in law is Victoria's 'Brodie's Law', which extended the offence of stalking in section 21A of the Crimes Act 1958 to cover behaviour that amounts to serious bullying (including cyberbullying).

The offence carries a maximum penalty of 10 years imprisonment, and can theoretically apply to children 10 years and older.
However, the negative consequences of wrongly criminalising young people under 18 have already been experienced in this space regarding 'sexting', and we would not like to see this repeated.

But perhaps a new Federal cyberbullying law, based on the current Commonwealth workplace bu llying definition.

could apply to both young people and adults w h ile including different sentence parameters.
That way, the same law would be recognised by all ages as defining acceptable and unacceptable behaviours across age spectrums, thus establishing a commonly understood standard of behaviour across Australian society.

Additionally.
Additionally.

a useful example in law can be found in New Zealand's Harmful Digital Communications Act 2015, which aims to 'deter, prevent, and mitigate harm caused to individuals by digital communications' and 'provide victims of harmful digital communications w ith a quick and efficient means o f redress·.

However, this Act does not define cyberbullying.

The law as part of an educational framework.

The law as deterrent.

The law itself is an educational tool.


Laws are in place to act as a deterrent and impact upon behaviours - to teach people that there are acceptable and unacceptable ways to behave.

This is further reason to have a nationalised standard legal definition of cyberbullying and to leverage the law to educate our community that such behaviour is unacceptable.

However, for this approach to have impact.


we need to be very clear and detailed on what cyberbullying is, how to identify it and ways to manage it and deal w ith it w ithin the continuum of social expectations.
The law benefits from programs like the Foundation's successful eSmart program that operates in schools and public libraries across Australia because it helps to remind institutions and communities of their role in maintaining policies and standards that keep children safe and protected and that they are part of the continuum of the law.
Alannah & Madeline Foundation 6 Our legal frameworks are also impacted upon by regulations like Victoria's Child Safe Standards and by legal obligations like Duty of Care that educational and other institutions that have responsibility for the health and well-being of children must abide by.
Alongside the obligation schools have to exercise a duty of care towards children, they also have a responsibility and are guided by curriculum and to educate children about respectful and appropriate behaviours online.

This includes informing children about the law and how it operates in relation to their role as citizens.

In this way the law itself is educational, and educational institutions have a role in educating children about the law.
Collaboration with law enforcement officers The Foundation has recently signed a Memorandum of Understanding w ith the Australian Federal Police, which we see as a vital step towards effectively addressing bullying and cyberbullying.

As we have learned from the experience of Brodie's Law, law enforcement officers require thorough training in all aspects of bullying behaviour, in addition to obtaining knowledge in the specifics of the law.
They also require an understanding of the social settings in which bullying takes place - namely schools, workplaces and social media platforms.

The Foundation is well placed to offer additional training in bullying behaviours and the various settings in which it takes place.

We have recognised that a whole-of-society approach to bullying and cyberbullying requires strong collaborations across sectors.

Any new or altered law must be supported by the resourcing of such training and collaboration.
Alannah & Madeline Foundation 7 Beyond the criminal code Addressing: a. other measures used to combat cyberbullying predominantly between school children and young people.
The Alannah & Madeline Foundation has a history of supporting schools and other community and learning institutions to: • • • combat cyberbullying strengthen community understanding and capacity to manage digital technology in ways that are safe and responsible in relation to children and young people reinforce the legal and community standards around technology and cyberbullying.
We have done this through: a. b. c. The National Centre Against Bullying eSmart Schools and eSmart Libraries The Digital Licence - especially our STARS Foundation partnership, working with Indigenous communities in the Northern Territory d. eSmart Workplaces.

Summaries of the first three initiatives aforementioned.

In partnership with McDonald's we have piloted Australia's first workplace cybersafety and bullying prevention program.
Cyberbullying is a phenomenon that commonly crosses over into workplaces and McDonald's are the first workplace in the country to achieve eSmart Status.

We see this as a vital development of the eSmart program, as it supports teenagers and young people who for the first time are having to balance their personal and professional lives.

We have identified key gaps that need to be addressed in improving the way we combat cyberbullying in our community.
These include: Economic cost of bullying In partnership with PwC, the Foundation has begun a project to establish the economic impact on our community of bullying and cyberbullying.
It will specifically estimate costs of bullying in schools in Australia, and will include: • • • calculation of the number of bullying incidents across Australia by type of bullying and student characteristics cost metrics and associated assumptions an economic model to calculate the impact of the cost metrics and estimate the total cost of bullying in Australia on a per annum basis.
Start early to change behaviour and culture The Foundation, within its strategic planning, has identified the need to support young children and parents.

We are embarking on a research project and developing new partnerships and projects to help us meet this identified gap.

Alannah & Madeline Foundation 8 Holistic approach Addressing: a. any other related matter.
It needs to be understood that the need for children and young people to be safe from violence will always require multi-faceted and complex approaches which requires the whole-of-community and includes legal, structural, technological and cultural solutions working together.
At the Foundation this is represented in the way we work and can be seen in the development o f programs like eSmart which supports whole communities to address cyberbullying and to foster pro social behaviours in our children and young people, so that they grow up to be citizens who respect and value the laws in place to address cyberbullying and any other aspect of digital citizenship.
Lesley Podesta CEO Alan nah & Madeline Foundation Alannah & Madeline Foundation 9 f r • ( r r I l L L l l L \.. esmart Sc o apshot s -valuation 2014 2015 Important aspects of the eSmart Schools framework for principals 98% the focus on student wellbeing • 98% embedding cybersafety across the curriculum - - - -~ • - 95% the development of explicit policies and guidelines• 92 % involvement of the whole school community• 9l% the explicit protocol s for teachers in modelling behaviours• SS% the establishment of Acceptable Use Agreements • Students feel safer: l. In terms of both bullying and cyberbullying 2.
Were aware of cyber risks 3.

Know what to do if an incident occurs Teachers feel more confident to: l. use technology 2.

Advise students on cybersafety 3.

Know what to do if an Incident occured School leaders say eSmart works to: 1.

Create a positive culture within the school 2.

Improve the way schools manage issues and incidents 3.

Prompts actions that would not otherwise have been taken vSmart Schools -· TIMAl.aMO~ iW -.. ~ UtlMad•...,.

F; .. a, ~ - .... ._.

.........


Prepared by: fya Universityy · Westem S;lr.ey &'~""All<'°@m·• .. ,, _ .. , .. ~ f.H~ ~...., w Executive umm 1.

Background r • • This evaluation What is eSmart Schools?

This report presents the findings of an evaluation of the implementation and impact of eSmart Schools, an Australia wide initiative of The Alannah and Madeline Foundation available to schools across Australia.
The Foundation for Young Australians led the evaluation, conducted over a two-year period (2013-2014) in Victoria, New South Wales and Queensland.

The research and evaluation consortium comprised the Foundation for Young Australians, the Institute for Culture and Society at the University of Western Sydney and the Pitt Group.
The purpose of the evaluation was threefold: 1. to understand the impacts associated with eSmart Schools 2. to discover how eSmart Schools is being implemented 3. to identify key areas for development.

The evaluation design drew from a program logic model1 and realistic evaluation• to assess the process, impact and outcomes associated with the flexible, ongoing implementation model of eSmart Schools (described in full in Appendix A).
This required accommodating complexity and diversity, and using a combination of methods to capture diverse data mapped to key research questions as presented in the eSmart Schools program logic.
The evaluation methodology comprised: > an environmental scan > surveys of the school community (526 parents, 2,956 students, 1,792 teachers, 617 eSmart Schools coordinators and 396 principals) in 2013 and 2014 and a survey of 23 non-eSmart comparison school principals in 2014 > case studies (seven diverse longitudinal and nine point-in-time) involving schools in all three states in different types of school settings.
Case studies were compiled from interviews with 29 eSmart Schools coordinators, focus groups with 125 staff, workshops with 243 students in Years 5/6 and 7/8 and contextual observations .

A world-leading initiative, eSmart Schools helps Australian schools manage cybersafety and deal with bullying and cyberbullying.
Similar to the SunSmart campaign, eSmart Schools uses a behaviour and social change approach to foster a community of students, teachers and parents to become smart, safe and responsible on line.

It supports schools to embrace the benefits of technology while reducing their exposure to cyber risks, such as cyberbullying, online sexual predation, sexting, identity theft and fraud.

Technology practices are subject to rapid change, and effective cybersafety strategies must be flexible to keep pace.
It is increasingly being recognised that long-term, flexible, holistic approaches to cybersafety offer the best way to empower our children and young people to grow up safe and well in a digital society.

New strengths-based approaches are shifting cybersafety strategies towards a 'digital resilience' framework, rather than a singular focus on risk and protection.
These strengths-based approaches recognise the importance of skilling users, not only to engage safely but also to ensure their online engagements maximise the full potential of connectivity.

Australian schools are now faced with a plethora of products, resources, applications, tools, software, programs and campaigns that address the issue of online safety from a variety of perspectives.
eSmart Schools provides schools with a framework to manage this crowded landscape and to select resources that suit their specific circumstances within a broader resilience approach (Figure 1).

Schools appoint a staff member to the role of 'eSmart Coordinator' who then leads implementation of eSmart Schools.
With funding from the Commonwealth Department of Education, Employment and Workplace Relations, The Alannah and Madeline Foundation piloted eSmart Schools in 159 schools across Australia in 2009- 2010.

It has now been rolled out nationally in almost a quarter of schools nationally, with Victorian schools initially supported with funding from the Victorian Government Department of Education.

........................................ ... .


.................................................................... .. .................................................. ..................................................... .

4 eSm:irt Sc~oo s Evaluat,on Repon 2015 Executive Summary: 2.

Impact eSmart schools are safer and more respectful There is strong evidence to suggest that.

as schools progress through the framework.

eSmart Schools delivers accumulating impacts.


More schools reported being cybersafe and having respectful cultures as they moved through the successive eSmart Schools phases (planning, implementing, sustaining).

In general, staff in schools acknowledged that eSmart Schools contributed to positive culture change in their schools: changing school-wide culture and behaviour, prompting action that they would not have otherwise taken and improving the management of issues.
The case studies show that eSmart Schools results in a shift in the approaches taken by schools from reactive approaches that prioritise technology risk management practices to proactive approaches that more effectively acknowledge the role technology plays in young people's lives and that embed technology in the curriculum and daily life of the school.
While eSmart Schools has a positive impact in all types of schools, the evaluation shows greatest success in primary schools and combined (primary and secondary) schools.

School leadership repor ted a positive perception of eSmart Schools (92%) and almost all would recommend eSmart Schools to others (98%).

Principa ls reported that they chose eSmart Schools because of its whole-of-school approach and because they believed it to be a clear, researched, recognised fram ework.
eSmart Schools assisted schools to navigate the complex field of online safety and the multitude of available resources, products and programs.

It assisted schools to develop and implement a strategy to address the complex issue of cybersafety.

No standard recording or reporting system for bul[ying incidents is in place in any of the three Australian states in this study.
However, this evaluation found that eSmart Schools prompted schools to either review their existing recording mechanisms or introduce new mechanisms.

Although the number of reported incidents did not change from 2013 to 2014 (with an average of four per school per semester), the number of schools recording information increased from 72.3% in 2013 to 83.8% in 2014.

Reporting systems varied widely across schools, however, and data are therefore not comparable.
The data from this evaluation suggest that incidence may be a less useful impact measure than resilience, or the ability of a sta ff and student body to identify, respond to and move on from adverse events.

eSmart schools audit their practice and implement new policies, curriculum and teaching practice These evaluation findings support a growing body of knowledge, which recognises that flexible solutions offer the greatest potential.
Coordinators reported that the framework's major strength was it enabled them to systematically review (or audit) their needs and existing activities and to prioritise actions to build on their existing strengths.

eSmart Schools delivers the most pronounced impacts when schools use it to initiate a process of reflection on the school's culture and approach to technology use, as a precursor to developing strategies for improvement.
Schools that critically reviewed their policies cited many examples of change and improvement in a range of practices .

............... ........................................................... ......................................... , ......................................................... .
...................................... , ............................................................................ ................................................................................................ .

There is strong evidence that eSmart Schools helped schools deliver improved policy, curriculum and teaching practices: > The majority of coordinators (84%) reported that eSmart Schools helped deliver improved policies and procedures.
The number of policy strategies in place increased between 2013 and 2014 and as schools progressed through the framework.

> The majority of coordinators (82%) reported that eSmart Schools helped embed smart, safe and responsible use of technology across the curriculum and that additions were made to the curriculum in those schools that had progressed through the framework.
The few case-study schools that prioritised curriculum in their implementation of eSmart Schools identified creative ways to use technology to strengthen teaching practices and to consolidate a positive school culture.

> More than 85% of teachers reported incorporating smart, safe and responsible practices into their teaching.

This percentage increased between 2013 and 2014, and more teachers incorporated teaching practices in those schools that had progressed through the framework.
Teachers in eSmart schools are more confident using technology and addressing cybersafety and cyberbullying Other research, described in section 1.4, shows that a teacher's confidence in their own digital skills is an important predictor of their ability to teach and manage cyber-related content and issues.
As schools progressed through the eSmart Schools phases and adopted strategies to improve teachers' digital skills, teachers reported increased confidence in their ability to advise students and to incorporate positive use of technology into classroom practices and learning.

The case studies demonstrate that when schools that prioritise teachers' digital literacy as a key pillar of implementation, they achieve positive impacts.
Nonetheless, just under a fifth of students reported that teachers never or rarely talked to them about aspects of cybersafety or strategies for positive use of technology.

This may be related to the third or so of teachers who reported still feeling not very confident in their digital skills and/or to those schools where technology was restricted, decreasing the opportunities for teaching about it.
More could be done to include student developed and student-led activities The latest international research suggests the best outcomes occur when young people are involved in the design and delivery of resources and activities, something the eSmart Schools framework promotes.
This evaluation found a growing interest by school leadership in this strategy, with around three quarters reporting that encouraging student presentations was important for generating a school-wide culture and behaviour change (principals 70.0% 2013; 77.4% 2014; coordinators 73.3% 2013, 79.3% 2014).

The percentage of schools that included student-developed activities increased by 22% from 2013 to 2014; however, only half the coordinators reported their inclusion.
Coordinators in those schools that had progressed through the eSmart Schools phases were more likely to report incorporating student-developed activities.

In the small number of case study schools where students played an active role in the implementation of eSmart Schools, there was a strong alignment between student behaviour, knowledge and attitudes and the kinds of values, policies and practices the school was promoting.
esmortsc 100.s.of{;.at: 7 a1scuss1on of values (e g. resRect, inclusion, valuin difference) • • • Exe utive Summary: 3.

Implementation ...................... ...... ... ............................................................................................................................. ................................... .......................
The effectiveness of eSmart Schools implementation depends on context Successful eSmart schools are characterised by five factors The capacity of the eSmart Schools framework to have a positive impact on schools depends on the quality of implementation, which in tum depends on the school context: school culture, attitudes of teaching staff and students, leadership support, coordinator's role and levels of resourcing.

Where support for the coordinator was inadequate, there was low visibility of values.
Where there was a culture of negative attitudes to technology or punitive responses to cybersafety issues prevailed, the impacts were not as great.

In addition, for a small minority of schools with particular characteristics, the framework was challenging to implement.

Schools with a high proport ion of vulnerable or marginalised young people and/or students with disability or schools with a history of violent or cyber incidents experienced challenges in tailoring the framework to meet their needs.
This is an issue because these groups of young people have been shown to be more at risk online.

Coordinators in schools with complex or special circumstances expressed a need for greater support to identify and apply strategies and resources to meet their school's unique needs.
The evaluation identified five factors that are critical to the successful implementation of eSmart Schools (see Figure 2): Factor 1: An eSmart school involves the whole school community Successful eSmart schools had leadership and teaching staff who were enthusiastic about eSmart Schools, a strong coordinator, an eSmart Schools committee, leadership support, and student involvement in solutions.
Factor 2: An eSmart school has strong values and a positive culture Successful eSmart schools had strong values regarding wellbeing and respect and a positive and open culture around the use of technology.

Factor 3: An eSmart school is supported by eSmart Schools resources Successful eSmart schools made good use of the eSmart Schools resources: the framework and on line tool, the training, the help desk, the newsletter and the resources accessible via the on line tool.
Most coordinators reported that the framework and on line tool were of very high quality and easy to use (satisfied with the framework 82.8% 2013, 82.3% 2014 and on line system 81.7% 2013, 73.3% 2014).

Factor 4: An eSmart school has high-quality tech positive staff Successful e-Smar t schools had high-quality staff who encouraged and modelled positive use of technology and ensured their digital skills were up to date.
Factor 5: An eSmart school extends strategies for smart, safe and responsible use of technology into the home Successful eSmart schools engaged with parents to ensure that school and home were correlated in terms of smart, safe and responsible technology use .

............................................................... ........................... ............................. ................. ......................................... ........................................ .
Leadership & teaching sraff ore enthusiastic about eSmort and support young people's smort, sofe ond responsible use of technology Rollout occurs olooaside o strong school voTuos fromework thot focuses on wellbeing ond re.sped School hos o committed cSrnort coordinator, positioned in o wellbcing role, with produtlive relolionships within the school community School hos o diverse eSmort committee informed by young people's views and e)(periences Stoff model posifo1e Uie of technology and continuously leom to mointoin their digital literoey The devcloj)l1W!nt of o cybcrsofcly strategy is tmderpinnod by youth P.<Jrlic:,potion ond a whole-of-community approach School hos o culture of promoting the positive potential of young people's technology use within and beyond lhe classroom School allocates adequate limn ond resources lo implemenlotion School approaches implementation as an opp(>l1unily to refled upon school culture, policies and prodices · ond lo develop strategies to improve them School engages p<iref'lfs and the brooder community in the school's doy to-doy life School hos o good working reloliond1ip with the Alannah and Madeline Foundation and other eSmart sc:hools Five steps that could improve the reach and impact of esmart Schools The evaluation identified five steps that could help more schools become eSmart schools (see Figure 3): Next step 1: Help schools to better manage the process Lack of time for staff to administer and implement eSmart Schools and limited funds for training were the biggest barriers to participation reported by principals and coordinators.
Around a fifth (21.2%) of coordinators also reported that the complexity of the task and navigating the vast and rich body of resources to which the framework connects were challenges that hindered progress through the framework.

Next step 2: Enhance the opportunities for schools to learn from each other Coordinators and staff are generally excited to learn about and share innovative practice.
Around half the coordinators (49.4%) reported that eSmart Schools encouraged schools to interact and/or share their cybersafety and student-wellbeing practices with other schools (17.4% said it did not).

More could be done for those schools that find it difficult.

Next step 3: Promote an explicit focus on wellbeing and digital literacy There was a perception that the framework deals primarily with cyberbullying.
This perception limited the impact of the eSmart Schools framework because it resulted in schools underestimating the scope of what they could do and fails to focus on the digital literacy of teachers, which is an important factor for success.

Although for some schools digital liter.acy of teachers improved as they progressed through the eSmart Schools framework, enhancing teachers' skills and literacy continues to be a significant challenge.
Next step 4: Engage parents to correlate the home world with the school world The domain that schools found most difficult to implement was 'parent and community partnership'.

Coordinators reported it was a challenge to engage parents, and only a third reported that their school had built effective partnerships with local organisations (26.7%i.
Next step 5: Consider how to tailor approaches and resources for schools with specific or complex needs The surveys and the case studies show that schools with complex challenges and diverse needs find the online tool less useful, and there do not appear to be comparable resources that target and support specific groups or needs .
........................... .... ........ ........................................ ......................................................................................................................................... .. 12 eSma•t Sr.hools Evaluatio-, I " r ,., I ' Next steps Enhonc:e implementation by providing examples, temploles and other resources to help schools worl< through the myriad of resources oo offer Provido funding for teacher release.
Being time-poor was the number one reason for difficulty with both training implementation Promo1e besl practice excimple$ of how schools in dirletent :;otlings hove engaged parents Oelermine specifk sc1s of rMOurces 1hat could be of volue to schools wi1h complex or specific needs A me<:honism could bo found lo help schools share mOfe of their successful strotegies, useful resourc&a and innovative prodic:es Cleorly communicate eSmort Schools is o holis1ic opproudi for wellbei1'9 and !

'lot solely onfi.
'lot solely onfi.

bullying Conlinuc to encourage leochors to improve their digital literocy School engages parent$ ond the broader community in the school's day, to-day life The Alonnoh ond Modeline Foundation continue to p1ovide leadership for consiitency and coordination of s1ote/lerrilory government approaches to ensure lhey ore youth-centred, focused broadly on wellbeing ond pO$ilive about technology Executive Summary: 4.
Recommendations ...........................................................................................................................................................................................................................
Effective implementation and enhanced impact in the promotion of safe, smart and responsible use of technology may be achieved by further development of the following aspects of the eSmart Schools framework: Promote the importance of teachers' digital literacy as part of the broader focus on wellbeing (including a positive view of technology) Strengthen eSmart Schools' focus on wetlbeing and promote this to schools literacy and confidence early in their eSmart Schools journey, as this is key to delivering enduring impacts.
> Encourage schools to improve teachers' digital > Emphasise and extend the promotion and prevention approach that underpins eSmart Schools' holistic approach to promoting young people's safety and wellbeing in a digital society by reviewing program elements to ensure eSmart Schools aligns with the latest research on success factors in whole of-school cybersafety programs.
> Actively promote the alignment of eSmart Schools with wellbeing and resilience frameworks to ensure schools do not see it as simply an anti-bullying checklist.

> Provide leadership to encourage consistency and coordination between state and territory education departments in their approaches to cybersafety policy to ensure they are youth-centred, focused broadly on wellbeing and positive about technology.
Adopt a resilience framework to evaluate success > Ensure teacher education and training incorporates evidence from youth-centred research and uses intergenerational strategies.

> Encourage teachers to have more values-based discussions with students about their online engagements rather than simply focusing on instrumental aspects of engagement (e.g.

privacy settings).
privacy settings).

Young people do not distinguish between the online and offline worlds and translate their moral, social and emotional values to the online space.

Examine and promote ways to involve students > explore resources and partnerships that emphasise and supper t schools to take a student-centred approach to eSmartSchools implementation and student-developed activities.
> Collate and promote case studies of examples > Identify resilience indicators, and promote them as of best-practice student involvement an evaluation measure for schools to use in addition to lncident reporting to reinforce positive concepts and provide better information on impact (e.g.

online participation, knowledge of risks and safety strategies, problem-solving and seeking help).
14 p • Examine ways to engage parents to correlate the home world with the school world > Recognise and support the role of parents as key allies in the uptake of eSmart Schools.

Explore strategies, such as online social networking, to shape parental attitudes and behaviours and empower them as advocates of eSmart Schools.

> Ensure education for parents includes evidence from youth-centred research and uses intergenerational strategies.
> Review the extent to which the expectations for broader community engagement in the 'parents and community' domain is realistic and useful.

Enhance implementation with funding > Maintain- and where possible increase-the grant that accompanies the eSmart Schools framework to ensure that schools can take up eSmart Schools regardless of context or circumstance, particularly those primary and government schools with low ISCEA rankings.
with opportunities for schools to learn from each other > Model ideai implementation scenarios and contexts to schools by providing them with case studies of successful implementation.

> Create mechanisms to help schools share more of t heir successful strategies, useful resources and innovative practices.

This should include the promotion of inter-school exchanges and activities that model and promote positive cultures of technology use.
> Develop school-to-school mentoring schemes and/or collaborative implementation planning processes that enable schools to learn from and support each other.

with more assistance > Enhance the planning and implementing phases by providing additional concrete strategies, templates and one-on-one assistance and advice.
Develop tailored solutions for schools that face complex challenges or have students with special needs > Provide additional support to assist schools with particular challenges to develop implementation plans that are responsive to their specific school context.

> Ensure eSmart Schools addresses the needs of vulnerable young people and makes specific provisions for young people experiencing marginalisation.
This could be achieved in a variety of ways, such as needs-based targeted support and resources, marketing and communication strategies, and a redesign of the on line tool to include a school self-assessment feature with tailored recommendations for specific strategies or resources.

> Develop strategies to foster the framework's successful adoption in secondary schools.
Enhance sustainability > Develop ways to identify ongoing achievements and to promote the value of eSmart Schools over the long term to schools in the sustaining phase.

Examples include: - A system of staged achievement over a long-term period similar to a frequent flyer program whereby schools would achieve, for example, eSmart Bronze (two years of successful implementation), Silver (four years), Gold (six years) and Platinum (ten years).
The achievement of each status would be tied to reaching particular goals, developing best-practice initiatives trialled in their school setting, mentoring schools that are in the planning and implementing phases and becoming a regional 'champion' of the eSmart Schools framework (as opposed to simply marking time passed in sustaining the framework).
- inter-school knowledge exchange that provides opportunities to engage with other schools in the sustaining phase to share insights and experiences, to workshop challenges and to showcase interventions via , for example, a series of workshops or a biennial conference.

- eSmart Schools champions program in which experienced schools work with schools that are in the early stages of the eSma rt Schools journey to promote the eSmart Schools framework and provide guidance in the crucial implementing phase.
- A small-grants scheme, which would make small grants available to schools to trial new e-Smart affiliated interventions that would draw on existing resources to target particular issues or contexts.

.......................................................................................... .............. ............................................................. ................................... ................... esni.1rtsc'loo s.org.a~ 15 r r I r I r f r I [ I I l l l. l L L \.
l eSmart Libraries Final Evaluation - Statistics at a Glance (• ' Impact in the library OL reported improved staff knowledge and confidence in how to be smart, safe and responsible online 84% are confident that the staff would be able to assist them if they came across something unsafe online Survey results demonstrate that eSmart plays a key role In supporting library users to be safer online.
Library managers know how to report and act when coming across something unsafe unline Library managers beUeved the eSmart Framework had changes library practices for the better Library managers self reported being confident in managing a serious online risk "I think it's a great program ... it's an opportunity for libraries to be part of the conversation about the digital world ... and also showcasing that libraries are just as much about digital literacy now as they are about print literacy:' Library industry representative Source: Independent evaluation Final Report: Western Sydney University published June 2017 "' k:J,IJ!K' Alannah & Madeline ''tt:_ Foundation In partnership with TELSTRA FOUNDATION For more information visit esmartlibrarles.org.au eSmart Libraries Final Evaluation - Case Studies r. ( Library user #1 (adult), Metropolitan library service Peter moved to Australia with his wife and children 18 months ago.
On arrival.

they had to quickly find a house to move into - which was a new property in a newly developed area.

and didn't have internet or a phone line installed.

It took months for this to happen, due to issues between the phone company and the landlord.

Peter's local library played a crucial role in their lives: all of Peter's job searches and job applications were done online and being able to get online in the library meant that Peter could find employment.
"It made an enormous difference to our lives ... it meant that we could lead a normal life whilst we waited for things to be installed at home, and starting work."

Peter's local library is an eSmart Library.

He has noticed some 'cyber smart' initiatives around the library, such as the Deep Freeze on the public computers, which deletes any previous user's data from the public computer once they have logged out; and a recent cyber safety promotional campaign displayed on banners in the library.
Peters experience ot ai.king for assistance with technology in the library has been positive.

HEvery employee here [is] always there [for you], you ask them something, they help immediately."

Library user #2 (adult), Metropolitan library service John is a regular computer user at his local metropolitan library service.

The library service has a large number of public computer terminals as well as free wi-fi access.
It promotes its online safety and digital literacy training programs through an event program and website.

John is new to computers: he has been teaching himself to use them in the library over the past two years.

He cannot afford to have a computer at home and has not been able to get his landlord to install an internet landline.

Free access in the public library has allowed him to start to learn about the internet and to develop his digital literacy.
He has recently purchased a wi-fi dongle and has a tablet, but he still visits the library regularly to access email.

John has not participated in any training programs.

He is not aware of any for his age group (26-59 years), although they do exist.

When he needs support.

which is often, he calls on the library staff, and, in particular, the IT staff.

He writes down the instructions they give him in order to improve his knowledge and to help minimise the number of times he has to ask for assistance.
John tries to be safety-conscious online.

He's conscious of having a strong password; thinks a lot about whether emails may be spam; and asks staff if he is unsure.

He will continue to use the internet in his public library for the foreseeable future.

The library service offers him not only free access, but also staff from whom to seek advice as he navigates the online space.
"The staff are really great here ... they have helped me out a lot.." "' t!1JW:' Alannah & Madeline 'fl: Foundation In partnership with TELSTRA FOUNDATION For more information visit esmartlibraries.org .. au eSMART LIBRARIES EVALUATION FINAL REPORT PREPARED FOR THE ALANNAH & MADELINE FOUNDATION Rebecca Morecroft Project Manager Institute for Culture and Society Western Sydney University Gina Tracas Data AnalysUSocial Policy Consultant Jen Sherman Associate Professor Amanda Third Principal Research Fellow Western Sydney University Harriette Richards PhD Candidate Western Sydney University Dr Michelle Blanchard General Manager Policy, Research and Programs SANE Australia Dr Emma Kearney Senior Research Officer Institute for Culture and Society Western Sydney University ISBN: 978-1-74108-446-7 Suggested citation: Morecroft.
R., Third, A., Blanchard, M., Tracas, G., Richards, H., Kearney, E., Sherman, J., 2017, eSmart Libraries Evaluation Final Report, Western Sydney University, Sydney.

© Western Sydney University 2017 This work is copyright.

Apart from any use as permitted under the Copyright Act 1968, no part may be reproduced by any process without prior written permission from the authors.
2 ACKNOWLEDGEMENTS The research team would like to thank the participating library industry representatives, library services, managers, staff and users for their time and insights.

We would also like to thank the Young and Well Cooperative Research Centre for its support of the evaluation from May 2014 to June 2016.

We also acknowledge Jacqui Kinder, Maree Davidson and Linda Barry at the Alannah & Madeline Foundation for their support in delivering the evaluation.
INSTITUTE FOR CULTURE AND SOCIETY WESTERN SYDNEY UNIVERSITY Western Sydney University is a modern, research-led metropolitan university established in the late 1980s.

The University nurtures a distinctive, high-impact research culture, committed to enhancing our region's cultural, economic, environmental and educational development; and is responsive to contemporary challenges in Greater Western Sydney and beyond.
westemsydney.edu.au/ics ALANNAH & MADELINE FOUNDATION The Alannah & Madeline Foundation is a national charity that aims to keep children safe from violence.

The Foundation was established in memory of Alannah and Madeline Mikac, aged six and three, who, with their mother and 32 others, were killed at Port Arthur, Tasmania, on 28 April 1996.

The Foundation cares for children who experience or witness serious violence and develops programs designed to help prevent violence in the lives of children.
Its National Centre Against Bullying (NCAB) is a peak body comprised of experts in the areas of bullying and cyberbullying.
amf.org.au 3 eSVIART UBRA.Rl ESEVALUA 110N ANAL REFORT .................................................... 2 AO<I\IOVVI...E[)(:E\l 8\ITS ......................................... .................................................. 3 EXIDJTIVE SJM MARY ....... .... ................................................................................. 5 INTm)UCTION ...................................................................................................... 8 BACKGROUND ...................................................................................................................................................... 8 EVALUATION DESIGN ................................................................................................................................... 10 STUDY SAMPLE ................................................................................................................................................ 11 INDIVDUAL PARTICIPANTS ....................................................................................................................... 11 DATA ANALYSIS ............................................................................................................................................... 13 OUT OF SCOPE ................................................................................................................................................. 13 CONTEXT .............................................................................................................................................................. 14 KEYANDIN<3S ........ ...... .............. ............... ........................ .................................... 17 OVERVIEW ........................................................................................................................................................... 17 AREA 1: IMPACT ............................................................................................................................................... 19 AREA 2: IMPLEMENTATION ...................................................................................................................... 31 AREA 3: PARTNER AND INDUSTRY OBJECTIVES ..................................................................... 34 AREA 4: PERCEPTIONS .............................................................................................................................. 38 CDNQUSON ........................................................................................................ 41 F&Ef4::3\JCES .............................. ........ .................................. ..... ............................ 43 4 EXECUTIVE SUMMARY "The process of working towards becoming an accredited eSmart Library is increasing our know ledge and attitudes ... " (Library manager, regional library service, Wave One) This report brings together the two waves of findings of the eSmart Libraries evaluation, undertaken between May 2014 and May 2017 by the Young and Well Cooperative Research Centre (Young and Well CRC) and Western Sydney University.
The evaluation engaged over 1000 library managers, staff, users and industry representatives, across 15 library services in surveys, interviews and focus groups.

Findings highlighted the excellent progress the eSmart Libraries initiative is making in supporting library services to enhance the digital literacy and on line safety of staff, users and the broader community.
The eSmart Libraries initiative was found to support participating library services to better respond to and develop library users' digital literacy and cybersafety needs.

This was ach ieved through building management and staff confidence, introducing or improving internet access policy and procedures, delivering training and support to library users, and engaging more broadly in the community.

Some key findings were as follows.


The eSm art Li br ar ies initiative was found to support participating library services to better respond to and develop library users' digital literacy and cyber safety needs.

Library management confidence and engagement are essential to the effective implementation of eSmart Libraries.

Across the two waves of data collection, an increased number of library managers reported improved knowledge and confidence in being smart, safe and responsible online.
In particular, an increased number of library managers self-reported knowledge of how to stay safe on line, protect privacy, report/act when coming across something unsafe, and uphold their responsibilities online.
Increased library management engagement in eSmart Libraries across the board represents an excellent outcome for the eSmart Libraries framework, because library services' readiness and capacity to implement eSmart Libraries is deeply influenced by its leadership and their capacity to support staff and engage in the initiative.

Library management confidence and engagement are essential to the effective implementation of eSmart Libraries.
The findings suggest th?d the AMF should continue to f dentify new strategies to foster local government and library management engagement; through peer support opportunities, presentations at local government and library sector forums, and via the advocacy of the eSmart Reference Group.

5 Library staff knowledge is essential to supporting library users' capacity for smart, safe and responsible online engagements.
Findings highlighted that enhancing staff capacity is critical in order to maximise the benefits to users of their on line engagements through library services.

Across both waves of data staff consistently emphasised the need for further upskilling in digital literacy and cybersafety.
Li br ar y staff know ledge is essential to supporting library users· capacity for smart , safe and responsible online en gag em en t s. Findings suggest there is a strong rationale for AMF's strategy of continuing to work with the library industry; to identify opportun ities for library staff to upskill in digital literacy and cybersafety and to collaborate in and cross-promote digital literacy and cybersafety training programs.

This could include utilising AMF Connect and eSmart Membership.
Effective implementation of the eSmart framework also depends on an ongoing conversation with libraries, industry and the AMF.

There was an increase in the number of libraries implementing the framework and reaching eSmart status over the period of the study.
Between the two phases of evaluation, the AMF began to address the challenges to impleme·ntation - including limited time and staff resources, lack of management buy-in and navigating complex organisational policy processes - all identified in Wave One.
Committed to continuous improvement, the AMF adapted strategies to address these challenges, including increasing face-to-face visits with library services to provide more one-on-one support to eSmart Coordinators; and streamlining libraries' interactions with the framework via the online system tool.

The evaluation indicates that the AMF should continue to seek opportunities to offer support to library services in both the implementation and sustaining phases of the framework.
In particular, the AMF could seek further opportunities to develop the eSmart Libraries network of libraries to promote the sharing of practical strategies to foster digital literacy and online safety between libraries.

Opportunities identified by this evaluation include: Effective implementation of the eSmart framework also d ep en d s on a n on go i n g conversation with libraries, industry and the AMF.
• utilising existing library network email lists to establish a community of interest or online biog; • developing a peer mentoring platform and accompanying resources to encourage knowledge-sharing between library services; • providing opportunities for library services to share their experience of implementing the framework (e.g.

, an annual conference or regular workshop series); • utilising AMF Connect and eSmart Membership options to provide ongoing resources and support.
6 Taking a targeted approach to engaging library users in eSmart Libraries will likely enhance its impact.

Evidence indicated that older Australians and adults with limited digital literacy were benefiting the most from the implementation of eSmart Libraries.

However, the work undertaken to implement the framework is largely unseen by library users.

As such, opportunities should be sought to raise the profile of the framework with library users.
To complement the offerings of eSmart Schools, the AMF could use eSmart Libraries to support young people transitioning from education to employment, culturally and linguistically diverse (CALO) young people, and/or young people outside of mainstream education settings.

In addition, the AMF could explore opportun ities to maximise exposure of the initiative with its funding and industry partners.
Strategies could include: Taking a ta r geted approach to engaging I ibr ary users in eSm a r t Libraries w i 11 Ii k el y enhance its impact.

• developing targeted messages and resources for digitally • disadvantaged user groups (including those seeki ng employment, and newly arrived and CALO community members); identifying ways to foster and maximise consistent messaging across multiple environments, building on existing programs and products (e.g., eSmart Schools and the eSmart Digital Licence).
Overall, the evaluation found that the eSmart framework enables libraries to take a systematic and comprehensive approach to enhancing digital literacy and cybersafety strategies in Australian libraries.

... the eSmart framework enables libraries to take a system a tic and comprehensive approach to enhancing d igital literacy and cybersafety strategies in Australian libraries.
Given both the rapidly evolving nature of on line and networked technologies and the transforming role of libraries, the evaluation recommends aiming for continual improvement by developing tools and processes to enable refinement.

To ensure the framework continues to meet the needs of the library industry and sector partners, evaluation strategies could include a regular pulse-check survey, and monitoring of feedback from libraries via an online eSmart Libraries knowledge-sharing community.
7 INTRODUCTION BACKGROUND The Alannah & Madeline Foundation (AMF) engaged the Young and Well Cooperative Research Centre (Young and Well CRC) and Western Sydney University to undertake an evaluation of the eSmart Libraries initiative between May 2014 and May 2017.

This included two point-in-time data collection waves (Wave One and Wave Two) to assess the initiative's impact.

What is eSmart Libraries?


eSmart Libraries is a socia l and behaviour change initiative developed by the AMF in partnership with Telstra Foundation.

It collaborates with Australia's library network to connect libraries and their users with tools and resources to improve cybersafety and wellbeing, and to faci litate digital inclusion in the community.

Behaviour change is a research-based approach that addresses knowledge, attitudes and behaviours or practices.
It is informed by a combination of theories including social cognitive theory, theory of planned behaviour and - best known - the trans-theoretical or stages-of-change model.

As an approach, it has long been used in combination with broader social change strategies to underpin significant health initiatives, such as Quit, SunSmart, and those of the Transport Accident Commission (TAC) and WorkSafe Victoria.

eSmart Libraries is grounded in a social and behavioural change communications framework (SBCC).
SBCC uses science and data, as well as creative ideas, in order to: • influence social norms to support long-term, sustainable behaviour change at a population level; • motivate shifts in behaviour and practice at individual, organisation and community levels; • encourage community action; • influence decision-makers regarding provision of services, policy and regulation, or legislative change.
In its design, development and implementation, the eSmart Libraries initiative has also drawn on the learnings from campaigns such as Quit, SunSmart, TAC and WorkSafe Victoria.

It was developed in response to concerns that an increasing digital divide is leaving some of our society's most vulnerable members lacking in the skills and knowledge to be smart, safe and responsible users of technology (AMF, 2012).

8 What does being an eSmart Library mean?


Being an eSmart library means staff and library users know how to guard against security and privacy risks online, download content in a legal and ethical way, research and reference information, and manage reputation and relationships in cyberspace.

How was the eSmart Libraries evaluation structured?

The evaluation of the eSmart Libraries initiative included two waves of data collection: • Wave One - March to July 2015; • Wave Two - March to July 2016.
Both waves of data collection aimed to gather evidence in four key areas as defined by the AMF: • Area 1 - Impact; • Area 2 - Implementation; • Area 3 - Partner and Industry Objectives; • Area 4 - Perceptions.

What does this report cover?

This report summarises the key findings of a comparative analysis across Wave One and Wave Two of data collection, to identify the most and least significant changes and the implications for further strengthening and refinement of the eSmart Libraries framework.
Through a series of case studies, the report evidences the ways that the eSmart Libraries framework is supporting capacity build ing in libraries, fostering libraries' ability to better support the digital needs of the broader community.

The report also situates the findings in the context of political and social changes, both nationally and locally; in order to identify ways to respond to and maximise opportunities to further enhance the positive impacts of the eSmart Libraries framework.
It sits alongside the Wave One Preliminary Findings Report (Morecroft, Blanchard, Third et al., 2015), the Wave Two Findings Report (Morecroft, Third , Blanchard et.al.

, 2017), an independent evaluation of eSmart Schools (2015) and the AMF's own data monitoring through the online system tool and pre- and post-implementation surveys.
The findings in a nutshell Overall, the evaluation found that eSmart libraries: • helps to embed a culture of positive technology use in participating libraries; • supports libraries to create tailored and effective policies and procedures; • • enables libraries to track their progress in 'becoming eSmart'.
facilitates libraries' access to quality, evidence-informed resources; 9 EVALUATION DESIGN The eSmart Libraries evaluation is grounded in a 'mixed-methods' approach, which incorporates both qualitative and quantitative methods of data collection.

This ensures an informative assessment of the framework's key strengths and areas for improvement and/or expansion.

It also helps inform the ongoing development of the eSmart Libraries framework over time.

Who took part?
Who took part?

Participating library services were based in the Australian Capital Territory, New South Wales, Queensland, South Australia, Victoria and Western Australia.

Across the two waves of data collection , the research team visited 11 library services in total, returning to three of the 11 services in both waves of data collection.
In addition to the 11 participating library services, the research team undertook further interviews with four other library services, bringing the total number of participating libraries up to 15.

During Wave Two the research team also undertook an intercept study in the seven participating libraries, in which a research team member visited each library service and, over a four-hour period, invited library users to complete an on line survey.
Research participants took part in one or more of three activities: an online survey, focus group or individual interview.

Wave Two research participants in the online library users survey were able to opt in to a lottery to receive one of ten $100 book vouchers; and focus group and interview participants were given a $20 book voucher.

Phases of eSmart Libraries implementation Library services were in one of three phases of implementation of the eSmart Libraries framework: 1.
Planning: focuses on establishing mechanisms within the service to implement the eSmart Libraries framework; reviewing and identifying gaps in policies and procedures; and assessing the knowledge and skills of staff and library users.

2.

Implementing: focuses on creating or renewing all relevant policies and procedures; and delivering training and development opportunities for staff, library users and the broader community to learn and engage with cybersafety and digital literacy.
At the end of this phase, a library is deemed to be 'eSmart'.

3.

Sustaining: focuses on actions to retain eSmart status, namely: evaluating impact; consistently reviewing library practices; and sharing successes with the eSmart community and beyond.

10 Across both waves of data collection, 11 of the 15 library services were in the sustaining phase.
Importantly, all seven libraries in Wave Two were accredited as eSmart Libraries; a significant difference from Wave One, in which only two of seven participating libraries were in the sustaining phase.

Overall findings should be interpreted in light of this distinction.

STUDY SAMPLE The eSmart Libraries evaluation methodology relied on voluntary uptake by library services.

Invitations to participate were sent to library services that were registered with eSmart Libraries across Australia.
Library services in the Australian Capital Territory, New South Wales, South Australia.

Victoria and Western Australia expressed an interest.

In Wave Two, further information was provided to library services in Queensland and Western Australia, inviting them to participate in the library manager and eSmart Coordinator interviews only; an additional four library services opted in.

The evaluation team worked with participating library services to promote the online surveys to library staff and users.
The survey was promoted and circulated via library services' social media, websites, posters and flyers.

Uptake was limited at some library services.

Feedback from library services highlighted that the survey was long (taking up to 30 minutes to complete) and required proficiency in English.

In addition, library services assisted the research team to set up focus groups and interviews with library management, staff and users.
In some instances the focus groups and interviews were well subscribed to; however, in some instances there was limited engagement, particularly by library users.

When viewing the evaluation findings, it is important to bear in mind that there were limited participant numbers at some library services.

PARTICIPANTS There were over 1,030 participants across the both waves of data collection.
Of these, 574 library users participated in focus groups, interviews and/or the online survey: • 14% were 25 years of age or under; • 40% were aged 26-59 years; • 35% were aged 60 years or over; • 11% unknown.
11 Table 1: Participant groups-Waves One and Two data collection Survey* 15 307 -•• 129 187 49 Participant group Librarv manaQers Librarv staff Library eSmart Coordinators Library users: seniors aaed 60+ Library users: adults aaed 26-59 Library users: young people aged 10-25 Local government representatives AMF eSmart Libraries team Library industry representatives (Wave Two only) Focus group Interview 17 84 62 32 25 17 10 8 8 12 3 4 * 64 library user survey participants did not provide their age.
** There were 15 eSmart Coordinators surveyed - They are counted under library Managers and Staff.

Note: Some participants have been captured twice as they participated in a survey and an interview or focus group.

12 DATA ANALYSIS Survey data was analysed using SPSS statistical analysis software.

Qualitative data was assessed via a thematic analysis of interview and focus group transcriptions.
The analyses and case studies presented here do not disclose the identities of libraries or evaluation participants who took part in the evaluation.

Case study participants have been given pseudonyms.

OUT OF SCOPE The following components of the eSmart Libraries initiative, undertaken by the AMF , were introduced in mid-2016: improved functionality of the system tool; • • more frequent visits to libraries • offering of individual support sessions to eSmart Coordinators.
While this document refers to these components , they were not measured as part of this evaluation.

13 CONTEXT The eSmart Libraries initiative has entered the market at a time of significant transformation in public libraries.

Since its introduction in 2012, the initiative has been working w ith over 1,000 public libraries across Australia to implement the framework.
During this time, a number of policy changes, programs and advocacy platforms have influenced the environment in which eSmart seeks to achieve impact.

Following are some of the key contextual influences: • Digital Hubs.
Commencing in 2012, this four-year initiative of the federal government was designed to increase digital literacy skills; help people learn how they might use the National Broadband Network (NBN) and other forms of internet connectivity ; and be more aware of online safety (Australian Government Department of Communications 2013).

Forty Digital Hubs were established throughout the country , including a number in public libraries.
Librarians reported that the NBN will be good for their communities; and, in particular, that the Digital Hub program provided a new way for libraries to connect with communities as a community hub and site of social capital.

• Telstra's Tech Savvy Seniors.

This program was introduced in New South Wales as part of the NSW state government's Ageing Strategy in 2012, in Victoria in August 2014 and more recently in Queensland.
It is delivered primarily through public libraries and offers sen ior community members the opportunity to undertake training to build their digital literacy and confidence.

Telstra provides public libraries with funding and resources to deliver the program which includes library user tra ining that complements the eSmart Libraries offerings (Telstra, 2017).

• Government funding cuts , restructures and reforms.
Over the past few years, libraries across Australia have been impacted by government funding cuts, organisational restructures and reforms.

These have included local government amalgamations and boundary changes in New South Wales, rate capping in Victoria and cost-cutting in Western Australia.

Local and state government changes have impacted the way libraries are staffed and the tasks undertaken by staff.
Importantly, these changes highlight the time pressures on staff, who are potential ly overworked while also meeting new demands such as the need for greater digital literacy skills {Li, 2015).

• Federal government changes to MyGov.

In 2013 the federal government began to roll out the MyGov platform, replacing the www.australia.gov.au platform.
Since this time, government services (e.g., the Australian Tax Office) have been integrated into this online platform and many services can now be accessed by users online.

Libraries reported being impacted by an increase in library users seeking support to navigate these online systems.

14 • GoDigi.
14 • GoDigi.

Launched in October 2014 as a partnership between lnfoxchange and Australia Post, this four-year digital literacy program has a focus on ageing community members, regional and remote communities, CALO communities, Aboriginal and Torres Strait Islander communities, and small business.
It has five key components focused on face-to-face and online learning: a network of partner organisations; an online platform; pop-up face-to-face learning events; a mentoring program; and its National Year of Digital Inclusion (GoDigi, 2017).

• Office of the Children 's eSafety Commissioner.

Established in 2015, the Office focuses on ensuring that young people have safe and positive online experiences.
It provides online safety education; a complaints service for young people experiencing serious cyberbullying; and an online content scheme that enables individual to report illegal content.

In 2016, Libraries ACT trialled the eSafe Spaces program; a joint initiative between the Australian Library and Information Association (ALIA) and the Office.
The program provides training to library staff in order to assist young people in the library to report a serious cyberbullying offence (Australian Government Office of the Children's eSafety Commissioner, 2017).

• Digital transformation agenda.

Established in July 2015, the Digital Transformations Office (OTO) was set up to oversee the transition of information and transactional government services to a digital government services delivery model (Hazlehurst, 2015, May 12).
The OTO aims to merge all existing government websites into one location under the 'gov.au' domain.

On 14 October 2016, the Coalition government announced that the OTO would be replaced with the Digital Transformation Agency (OTA), with an expanded portfolio "overseeing the rollout of improved capabilities to users of government services" (Mclean, 2016).

• Smart Cities Plan .


The federal government's 2016 Smart Cities Plan highlights the potential of digital communications to "revolutionise the way governments engage with communities in the development of metropolitan and local plans and services" (Commonwealth of Australia, 2016, p. 27).
The Smart Cities Plan recognises the value of public data; and how anonymising and sharing data may assist government, industry and the community to better understand end-user needs and plan in ways that are citizen-focused.
Digital inclusion and commun ity engagement are not mentioned in the Smart Cities Plan, however if public libraries can position themselves as digital communication hubs linked to significant community infrastructure and urban renewal, they could potentially benefit from integrated planning activities around infrastructure and smart technologies.

• ALIA advocacy.

In 2016, ALIA released its report, "The Library and Information Agenda 2016", an agenda platform for the 2016 federal election.
The report highlights the need to better resource public libraries as the federal government rolls out its digital transformation agenda; as 15 "library and information professionals have the skills to support the drive to egovernment" (Australian Library and Information Association, 2016, p. 5).

Boosting this capacity is particularly important for those in the community who remain digitally excluded.
These policy and programmatic changes highlight the interest in digital literacy and cybersafety; and the important role public libraries play as digital hubs for local commun ities, actively contributing to improved digital participation by provid ing access and education programs.
The combination of locality with free access to devices, connectivity and education uniquely positions public libraries as spaces where digital participation can be promoted and supported (Jaeger, Bertot, Thompson, Katz & Decoster, 2012).

The role of libraries as key brokers for community access and education in a rapidly changing digital environment, particularly regarding digital government services, is now well established (Bertot, Real & Jaeger, 2016).
Libraries are recognised as uniquely located community spaces where a diverse range of users might seek assistance to access and learn more about digital resources.

Libraries are uniquely positioned to work in partnership with other community organisations and government agencies to strengthen community outreach programs that support digital literacy and inclusion in hard-to-reach community groups (Bertot, 2016; Fitzgera ld, Fletcher & Kop, 2016).
Simultaneously, governments can benefit from the established relationships public libraries have with their communities; as these existing connections bolster opportunities for community consultation activities and user-centred service design.

Responding to change Alongside the changes outlined above, the AMF has refined the eSmart Libraries initiative in response to sector feedback, evaluation findings and its own data collection.
As a consequence, it has introduced changes to the online system tool and framework in order to streamline its application; and has offered more opportunities for one-on-one support to library services, increasing library visits.
The AMF has also implemented key recommendations identified by the Wave One and Wave Two evaluation reports, including: • online professional development for library staff; • media campaigns with Telstra regarding the benefits of 'digital-savvy' libraries; • advocating for eSmart Libraries to also be eSafe Spaces; • integration of eSmart schools, libraries and the Digital Licence under one website; • development of AMF Connect a fee-based service providing a range of workshops and seminars on cybersafety, bullying and the impacts of violence on children and young people.
The workshops run for one hour and can be tailored to a library service.

Each workshop can also be tailored to audiences including children, adults, parents, employees and the general community.

introduction of an eSmart Membership for accredited library services is an annual subscription for accredited eSmart Libraries, offering a range of further resources and benefits to assist continued improvement.
• 16 KEY FINDINGS OVERVIEW Across both waves of data collection , library management and staff reported that they valued and supported the contribution of the eSmart Libraries framework; which enhanced libraries' capacity to respond positively to their users' digital needs.

Management and staff also reported that the framework domains were important in generating library-wide cultural and behaviour change.
In particular, library managers reported that the framework was adaptable to their library services' needs, indicating that the AMF has successfully addressed some challenges identified in Wave One regarding the perceived complexity of the framework.

Library services in Wave Two were three times more likely to have completed the framework and reached the 'sustaining' phase (Wave One = 30%; Wave Two = 100%).
Pointing to the success of the eSmart libraries framework, the following findings detail how library management and staff increased knowledge and confidence in ensuring their libraries were smart, safe and responsible.

... library management and staff increased knowledge and confidence in ensuring their libraries were smart, safe and responsible.

Across the two waves of data collection, an increased number of library managers valued and were actively engaged in the implementation of eSmart Libraries.
This is an excellent outcome: library leadership support is essential in order for libraries to achieve eSmart accreditation and successfully sustain the framework.

...

I i br ary management vat ued and were actively engaged in the implementation of eSmart Libraries.

Library management and local government representatives reported that the implementation of the framework has enabled them to demonstrate to their local government their expertise in digital literacy, cybersafety and online risk mitigation.
This indicates that the eSmart framework is playing a crucial role in increasing the visibility of coordinated approaches to fostering smart, safe and responsible online engagements via libraries.

Across both waves of data collection, library staff reported that there is increased demand to support individual library users with their digital needs.
There is a strong shift in library services away from the traditional transactional relationship; however, demand for one-on-one support to library users with limited digital literacy skills is placing additional pressures on staffing resources.

Library staff consistently emphasised the need for further upskilling in digital technology, to enable them to service library users' requests for support and guidance.
17 Collectively, these findings indicate that the eSmart Libraries framework is playing a critical role in library services' capacity to enhance, support and guide digital literacy and cybersafety education.
18 AREA 1: IMPACT Impact on library managers " [ eSm a r t L i b r a r i es h as] g i v en u s st r u et u r e a n d form and it's given us a focus and it's given us confidence .... the confidence to be able to say this is what we do and this is how we do it and this is why wedo it and we're looking after this for the CO m m U n i t y . "
(Li bra r y m an a g er, metro po Ii tan Ii bra r y service, Wave Two) Across Waves One and Two of data collection there was an increase in the number of library managers who reported knowledge and confidence in being smart, safe and responsible online.

Findings highlighted the importance of management's engagement in order for the eSmart Libraries implementation to be successful.

Particular findings were as follows.

• Digital knowledge.


An increased number of library managers in Wave Two self-reported improved digital safety knowledge as an outcome of the delivery of eSmart Libraries.

This included knowledge of: how to stay safe online; how to protect privacy; how to report/act when coming across something unsafe (Wave One= 50% , Wave Two= 83%); and how to uphold responsibilities on line (Wave One = 61 %, Wave Two = 100%).

• Online risk management.


Importantly, improved digital knowledge was seen to be translating into library managers' capacity to manage online risks in their work with users.

In Wave Two, library managers self reported higher confidence in managing cybersafety risks, with 100% stating they were confident/very confident in managing a low-level risk; compared to 71 % in Wave One.

Further, 86% self-reported being confident/very confident in managing a serious risk.

compared to 56% in Wave One.


This highlights the positive impact of the eSmart initiative on management practices in libraries that have attained eSmart accreditation.

• Engagement with eSmart.

Library management demonstrated increasing familiarity and support of the eSmart initiative across the two waves of data collection.

Whereas.

in Wave One.


not all library 19 managers reported being familiar with all components of the initiative (attributing this to staffing and management changes, and organisation restructures), in Wave Two, library manager interviews highlighted a strong level of engagement and interest in the initiative.
In Wave Two, library managers also reported that the framework was enhancing library services' work, guiding them via a systematic approach to internet user access policies and online procedures; staff and library user training; and outreach in the community.

"[eSmart Libraries is a] great catalyst for change for us, around digital skills and thinking about some of these issues."

(Library Manager , metropolitan library service, Wave Two) • Policies and procedures.
In Wave Two there was an increase in the number of library managers who reported that the eSmart Libraries initiative was impacting to improve policies and procedures as well as staff and library user knowledge.

(The number of library managers reporting benefits directly related to eSmart increased by 13% overall).
In addition, across the two waves of data there was a 26% increase in the number of library managers who believed the framework had changed library practices for the better (Wave One= 57%; Wave Two= 83%).

... the in er ease in engagement in eSmart by library management across Wave One and Wave Two represents an excellent outcome for the eSm art Li b r a r i es fr am ew or k .
A library service's readiness and capacity to implement eSmart Libraries is deeply influenced by its leadership and their capacity to support staff and engage in the initiative.

As such, the increase in engagement in eSmart by library management across Wave One and Wave Two represents an excellent outcome for the eSmart Libraries framework .

These results thus bode well for eSmart Libraries' capacity to produce flow-on benefits for library staff and users.
20 CASE STUDY: SMALL REGIONAL LIBRARY SERVICE This rural library service, with a small staff group, supports a population of over 36,000.

Approximately 30% of the community are library members.

The library manager took the lead as eSmart Coordinator, as the service was small and they did not want to increase the already-busy workload of one of their branch staff members.

The library service was an eSmart-accredited library in both waves of data collection .
Initially, the eSmart Coordinator established a working group with members from across the local council and developed an in-house cybersafety training package for all library staff to complete.

They also developed an induction pack for the library's team of casual shelving staff.

Staff reported increased confidence after completing the in-house training and were reported as having been proactive in providing tech-based support to library users.
The service has embedded eSmart into staff position descriptions, training and information packs for new staff.

It has also embedded cybersafety messages in the training and support delivered to library users, the library website, social media, local media and the library outreach service.
Some of the initiatives undertaken include: delivering tech activities at secondary-school career days and to young people in the library; partnering with local eSmart primary schools on displays in the library; offering tech help sessions to build staff confidence in working together; delivering Telstra's Tech Sawy Seniors program; offering the Broadband for Seniors mentoring program; partnering with GoDigi to launch the National Year of Digital Inclusion; delivering information sessions with cybersafety experts.
Findings highlighted an increase irll staff confidence between the two waves of data collection.

Library management and staff also highlighted the importance of continuing to build staff confidence through refresher training and reminders.

The library recently managed a cybersafety incident in the library; which demonstrated that staff were able to enact the library service's improved policies.
Staff reflected positively on the importance of having a procedure in place and supporting each other in implementing it.

The library has demonstrated exceptional leadership, and, through its library manager, continues to be a passionate eSmart advocate, mentoring another rural library service to implement the framework.

21 Impact on Library staff " I 'm mu eh more rigorous and security -conscious than I was before."
(Staff member , rural library service, Wave One) The evaluation results show increases in overall staff buy-in to the eSmart Libraries framework.

Wave Two library managers reported an increase in staff who had been 'willing participants' in the eSmart Library journey (Wave One = 67%; Wave Two= 83%).

Across the two waves of data collection , library staff, like library managers, reported increased confidence and knowledge in how to be smart, safe and responsible online.
Findings highlighted the importance of library staff being upskilled in digital literacy and cybersafety in order to continue to support library users' safe use of technology in the library.

... library staff.. .reported increased confidence and knowledge in how to be smart , safe and responsible online • Staff competency and confidence.
An increased number of staff reported that they felt more 'competent and confident online to support library users to use technology in the library'; with a 10% increase from 76% in Wave One to 86% in Wave Two.

Staff feedback showed that they had a good grasp of the basic technology skills needed to help library users access technology in the library.
This was particularly evident across the three library services visited in Wave One and Wave Two, with increased numbers of staff at all three libraries reporting improved confidence.

This result was also validated by 100% of surveyed library management, who reported that that staff were provided with information to read about being smart, safe and responsible online.

• Management confidence in staff competency/confidence.
Library staff's self-assessments were corroborated by library managers' assessments of staff capabilities: in Wave Two there were marked increases in the percentage of library managers who thought their library colleagues felt competent/confident online (Wave One = 50%; Wave Two = 83%); and who reported 'improved staff knowledge and confidence in how to be smart, safe and responsible online' as a key benefit of implementation (Wave One= 86%; Wave Two= 100%).
Only one manager was 'unsure' about the impact of eSmart, saying staff had a reasonable amount of knowledge before the framework was implemented.

22 • Managing online risks.

Results in Wave Two showed increases in overall confidence levels amongst staff for managing both 'low-level' (Wave One: 47%; Wave Two = 59%) and 'high level' (Wave One = 35%; Wave Two= 48%) cybersafety risks in the library.
Across both rounds of data collection a large number of staff (Wave One: 20%; Wave Two: 30%) reported that they were 'undecided' regarding their confidence in managing 'serious cyber-safety incidents'.

Whilst it might be desirable to see increased staff confidence in managing serious online risks, the latter finding does not necessarily represent a failure of the eSmart Libraries framework.
It is possible that, as library staff's digital literacy has increased, they have simultaneously become more aware of risks and the complexity of handling serious issues.

Individual library staff should not be handling serious on line risks without the guidance of their library managers.

As such, the ambivalence library staff expressed in this regard was appropriate, as long as library staff are aware of the procedures for seeking appropriate support and guidance if they encounter serious risks. "
Previously, if[Manager] had said let's implement t ech t i m e on [a) M on day m or n in g , w here anyone can ask anything and anybody would help them, the staff w ou Id have had a I itt le panic attack out theback ... whereaswhat I saw when [Manager] said w e are going to have t ech time it w i 11 be at 10 : 3 0 am ... ever y one said oh y ea h that's a great idea ... " (Staff member, rural library service, Wave One) • Staff training.
eSmart appears to be having an impact on staff training: more library managers and staff reported that staff had participated in train ing (Managers-Wave One= 80%, Wave Two= 100%; Staff Wave One= 30%, Wave Two= 50%).

Wave Two findings highlighted that staff training was most commonly delivered by a library colleague, rather than an external agency or the AMF.
Across both waves of data collection, staff and management consistently reported the need for ongoing and refresher training in cybersafety and new technologies, due to the demand on the library service.

Furthermore, they indicated that this should remain a key focus of eSmart Libraries implementation for libraries, whether they are just commencing their eSmart journey or in the sustaining phase.
The AMF has developed an eSmart Membership model and AMF Connect (a consultancy service) that libraries may mobilise to assist in upskilling staff.

23 CASE STUDY: LARGE REGIONAL LIBRARY SERVICE This busy regional library service delivers a large range of programs through its nine branches.

The service experiences high rates of computer and wi-fi use.

The eSmart Coordinator for the service is a senior staff member.
The library service has a large staff group who, in the initial stages of the implementation of eSmart Libraries, reported limited digital literacy skills overall.

When the research team first visited this library service, the service was halfway through the implementation of eSmart Libraries, but had stalled due to both internal restructuring and the challenge of bringing the working group together amid competing priorities and limited time.
By the time of the research team's second visit, the library had reached eSmart accreditation.

The restructure now allowed the delegation of tasks by the eSmart Coordinator to other staff members, facilitating new momentum.

With a large staff group, ensuring the upskilling of staff has been both a challenge and an opportunity for the library service.
Initial approaches to upskilling staff included : providing staff with digital devices to explore in staffrooms; upskilling staff by 'buddying up' on training ; awareness-raising sessions in staff meetings; recruiting new staff with digital literacy skills.
In the 12 months leading up to Wave Two data collection, the service has had a stronger focus on upskilling staff in digital literacy through the implementation of a new staff training model; in which the Digital Services Team trained a small but core staff group in cybersafety and digital literacy skills.

This staff group was then rostered to visit different library service sites to offer users and staff guidance and advice.
At the second evaluation visit, this new approach was still in its infancy and the service was still working through how to further build staff capacity.

However, Wave Two findings highlighted a slight increase in staff who reported reading information about being smart, safe and responsible online.
The second evaluation visit also highlighted an increase in numbers of staff who felt competent and confident online; were able to support users to use the technology in the library; and reported that their colleagues felt confident online.

The library service had also received NBN Digital Hub funding for one of its branches.

With the wrap-up of this funding it integrated its digital hub activities across the service.
Digital services offered to library users now include: one-on-one digital literacy support; drop-in tech sessions; how-to sessions on using e-books, tablets and smartphones; cybersafety sessions.

Wave Two findings highlighted a 19% increase in library users who reported feeling supported to manage online risks in the library.
The library service will continue to use the eSmart Libraries framework to trigger its ongoing review of approaches, policies and procedures regarding the promotion of cybersafety and digital literacy across the service.

The service is keen to mobilise its working group to focus on broader community engagement.

24 Impact on Library users "I feel safer onlinejust because of the course I've done [through the library]."
(Adult library user, rural library service, Wave Two) Direct impacts on library users are difficult to attribute specifically to eSmart, as the work undertaken through the framework is largely unseen by library users.

The evaluation shows that enhancing library staff's capacity to support users to access technology in smart, safe and responsible ways significantly enables users to maximise the benefits of their online engagements through library services .
Strategies for ensuring the best user outcomes include staff training; policy and procedural changes; and targeted digital literacy and cybersafety activities branded as eSmart.

Find ings were as follows: • In-library cybersafety initiatives.
Between Waves One and Two, greater numbers of library managers and staff reported an overall increase in cybersafety initiatives being undertaken by libraries, as outlined in Table 2. library ... enhancing staff's capacity to support users to in smart, access technology safe and responsible ways significantly enables users to maximise the benefits of their online engagements through library services.
Table 2: Cybersafety initiatives undertaken by libraries, as reported by managers Cybersafety initiatives Training for the public (groups and/or individuals) Visual advertising (e.g., cybersafety posters/brochures) Cybersafety 'help button' on all computers Guest speakers on responsible internet use and cybersafety Library website links to cybersafety awareness information Staff training in cybersafety and digital literacy Targeting children (e.g., free train ing, information sessions, parent consent forms) Internet Terms and Conditions of use Other (e.g ., networking) % of respondents Wave One (N = 7) 100 Wave Two (N = 8) 100 43 14 43 57 57 0 14 43 100 43 43 86 100 43 100 14 25 • Visibility of cybersafety initiatives .
In Wave Two, library users reported a slight increase (9%) in the visibility of cybersafety initiatives withil') libraries; and 17% more library users reported seeing a poster and/or other media about eSmart in the library, demonstrating that libraries in the sustaining phase are gradually raising the profile of eSmart.
Table 3: Cybersafety initiatives in the library, as reported by library users Questions regarding cybersafety initiatives % of respondents Wave One Wave Two (N = 235) (N = 141) Have you seen any advertising for cybersafety training in your library?

Have you seen any posters/media about eSmart in the librarv ?

Over the past year, have you noticed any changes to how your library provides internet services?

41 30 28 43 47 35 • User willingness to approach staff for cybersafety help.
There was a 12% increase in the number of library users who said they would ask staff for assistance if they came across something unsafe online at the library (Wave One= 72%; Wave Two= 84%).

The number of library users who said they would not ask for assistance remained consistent across both rounds of data collection; with this cohort reporting, for example, "I am confident enough to deal with the issue myself'.

• Cybersafety information sources.


As Table 4 indicates, library users reported that they accessed a variety sources for information and education about cybersafety.

Users' reliance on the library as an information source was reasonably consistent across both waves of data.

Table 4: Library users' cybersafety information sources Cybersafety information sources Media Work/colleagues Friends Library School Home (e.g.
, parents or other family) Other* % of respondents Wave One (N = 278) Wave Two (N=151) 27 21 12 10 6 1 12 16 17 16 12 19 7 7 26 *Other: personal computing experience or IT training, online research , computer magazines, the general community, or a mix of sources.

Note: an increased number of young people responded to the survey in Wave Two of data collection.

This likely accounts for the increase in numbers citing school as an information source.

• Diversity .
• Diversity .

Across both waves of data collection, participating library services highlighted the diversity of library users requiring access to technology.

In particular, library managers and staff discussed the need to support library users with limited digita l literacy skills, including seniors, people seeking employment, and newly arrived and/or CALO community members.
Library services are one of only a few community spaces whe re community members with limited access to technology can gain access to both technology and support.

The AMF could consider providing targeted strategies and resources to meet the needs of these groups through the eSmart Libraries framework.

27 Case studies: Library users' access to technology LIBRARY USER #1 (ADULT), METROPOLITAN LIBRARY SERVICE John is a regular computer user at his local metropolitan library service.
The library service has a large number of public computer terminals as well as free wi-fi access.

It promotes its online safety and digital literacy training programs through an event program and website.

John is new to computers: he has been teaching himself to use them in the library over the past two years.

He cannot afford to have a computer at home and has not been able to get his landlord to install an internet landline.
Free access in the public library has allowed him to start to learn about the internet and to develop his digital literacy.

He has recently purchased a wi-fi dongle and has a tablet, but he still visits the library regularly to access email.

John has not participated in any training programs.

He is not aware of any for his age group (26-59 years), although they do exist.

When he needs support, which is often, he calls on the library staff, and, in particular, the IT staff.
He writes down the instructions they give him in order to improve his knowledge and to help minimise the number of times he has to ask for assistance.

"The staff are really great here ... they have helped me out a lot..." John tries to be safety-conscious online.

He's conscious of having a strong password; thinks a lot about whether emails may be spam; and asks staff if he is unsure.

He will continue to use the internet in his public library for the foreseeable future.
The library service offers him not only free access, but also staff from whom to seek advice as he navigates the online space.

28 LIBRARY USER #2 (ADULT), METROPOLITAN LIBRARY SERVICE Peter moved to Australia with his wife and children a year and a half ago.

When they arrived, they had to quickly find a house to move into.

The house they rented was a new property in a newly developed area of the city, and didn't have internet or a phone line installed.
It took four and a half months to get a phone line, due to issues between the phone company and the landlord.

Peter and his family found the local library and started visiting regularly to access books, DVDs and the internet.

The library has played a crucial role in their lives.

All Peter's job searches and job applications were done onl ine.

Being able to get online in the library meant that Peter could find employment.

"It made an enormous difference to our lives ...


It meant that we could lead a normal life whilst we waited for things to be installed at home, and starting work ... " Peter's local library is an eSmart Library.

He has noticed some 'cybersmart' initiatives around the library, such as the Deep Freeze on the public computers, wh ich deletes any previous user's data from the public computer once they have logged out; and a recent cybersafety promotional campaign displayed on banners in the library.
Peter's experience of asking for assistance with technology in the library has been positive.

"[E)ach and every employee here [is] always there, you ask them something , they help immediately ... " 29 LIBRARY USER #3 (YOUNG PERSON), METROPOLITAN LIBRARY SERVICE Sara_h is a 19-year-old who has been visiting her local metropolitan library service with her sister since she was in prep.

Sarah uses the library service to borrow books and to study.
Until a couple of months ago she didn't have the internet at home; and until she started university a year ago she never had a computer.

All her primary and secondary school study was undertaken at the library, where she could access the computer and internet free of charge.

For example, she would use the computers to work on her chemistry practical assignments and geography reports, and she would print off any slides or reports required by teachers.
The library service also promoted local support services and , through her engagement with the library, Sarah learnt about and engaged with the youth services programs in her area. "

I would not have done half the things if it wasn't for that.

· The library service that Sarah visits has almost 50 public-access computers and delivers a range of digital literacy programs.

The service became eSmart in 2016.

It promotes its eSmart Libraries status through the website and within the physical library.
It has recently started to run Telstra's Tech Sawy Seniors program in different community languages.

Sarah is one of many students regularly using the library space for study.

This library is broadly considered to be a safe place in the community for families to go.

Reflecting this, Sarah considers the library to be a safe place; in particular, regarding accessing the internet.

"[A]t home you can hide in the corner and nobody can see what you are doing.
But in a public library, people are walking by, there are librarians .

It is a really open space."

She believes the library is a safe place for children.

"I'd pretty much guarantee safety for kids, it's a good environment and the librarians are pretty vigilant about what's happening."

Sarah values the library, as it has played an important role in her access to technology for education , as well as reading for enjoyment.

"I do not want the library to go away.

I need it here forever."


30 AREA 2: IMPLEMENTATION "eSmart underpins all that we do with technology; the message is reinforced through training.

Our position descriptions for staff all include an eSmart statement.

Our customer charter includes an eSm art statement.

We have thought about how best to [promote] the eSmart message through outreach activities and resources."
(eSmart Coordinator, regional library service, Wave One) The AMF's rollout of the eSmart Libraries initiative saw an increase of registered library branches from 361 in 2014 to 1,056 as at May 2017, with 313 individual library branches reaching full eSmart accreditation.

This represents a significant achievement in scaling the framework and positioning it for maximum impact across the sector.
All seven library services that took part in Wave Two data collection had reached eSmart accreditation and were in the sustaining phase of their implementation.

As such, in Wave Two there was reduced focus on the challenges to implementation, compared to the Wave One interview and focus group findings.

Specific findings were as follows.

• Ease of implementation - library managers.


Importantly, as mentioned above, in the Wave Two survey findings, no library managers reported experiencing challenges to implementing eSmart; compared to more than half of respondents in Wave One.

Whereas Wave One library managers described the initiative as, for example, "a very involved and challenging process" and rated this as their number-one learning, in Wave Two this issue was rated last and not considered a key learning.
Th is is likely because library managers had, in general, worked through the eSmart Libraries framework and were therefore more familiar and comfortable with it.

• Ease of implementation - eSmart Coordinators.

Showing a similar trend, survey findings highlighted that reduced numbers of eSmart Coordinators in Wave Two (6%) experienced challenges in implementing the eSmart framework.
Challenges that were consistently reported by eSmart Coordinators across both waves of data included lack of time to dedicate to implementation; limited staff resources dedicated to implementation; management challenges such as a lack of buy-in or willingness to allocate time; and complex organisational policy processes.

31 • Management engagement.

As mentioned in Area 1, there was an increase in management engagement in the eSmart initiative.
In addition to the statistics reported above, in Wave Two library managers also reported increased engagement in eSmart working groups (Wave One = 43%; Wave Two= 67%); and accessing online eSmart resources (Wave One= 50%; Wave Two= 60%).

Libraries that had successfully reached eSmart accreditation across both waves of data showed increased management engagement in the initiative.

• Participation in eSmart.


An increased number of library staff participated in the eSmart Libraries initiative in Wave Two (Wave One = 75%; Wave Two = 88%).

Implementation appea red to be translating into greater staff awareness and capacity to support users' digital engagements.

Greater numbers of library staff reported that they had accessed online resources (Wave One= 48%; Wave Two= 71%).
There was also a slight increase in the number of staff who had read information about being smart, safe and responsible online (Wave One = 62%; Wave Two = 68%) and implemented what they learnt within the library (Wave One= 47%; Wave Two= 53%).

• Staff training .

A key find ing was the importance of upskilling staff to support library users' cybersafety and digital literacy education.
Overall, the results show that one third of all library staff had undertaken training as part of their involvement in the eSmart Libraries initiative in Wave One, with this percentage increasing to more than half in Wave Two (51 %).

This indicates that eSmart is assisting libraries to prioritise staff training, which is an important component for delivering on library users' needs.
A majority of staff reported that the training had met their needs (Wave One= 75%; Wave Two= 71%), demonstrating that the eSmart training components were meeting expectations and provided relevant training.

Library services, and particularly larger services, face challenges in upskilling large staff groups, due to rapid technological change.

Some libraries had been highly successful in engaging their staff in training and related capacity-building activities.

• Communication with the AMF.


eSmart Coordinators reported strong levels of satisfaction with the AMF's communication processes across both rounds of data collection.

In response to the recommendations arising from Wave One of data collection and analysis, and sector and library-services feedback, the AMF increased its engagement with registered library services to support implementation of the framework.
The AMF reported that its strategy had been to increase visits to library services; to provide more one-on-one support to eSmart Coordinators; and to streamline the framework via the online system tool.

32 • Cross-sector communication .

It is worth noting that AMF reported that they could benefit greatly from hearing stories from libraries that have been successful in implementing eSmart, in order to help scale opportunities across the sector.
Comparative findings across Waves One and Two show that there has been positive progress, with successful implementation of the framework correlating with increased management and staff engagement.

This highlights the importance of active engagement in the rollout by library management and staff.

Implementation of the framework will continue to be an ongoing challenge for library services, due to competing priorities and requirements.
As such, the AMF will need to reflect on its ongoing approach and engagement with library services to support implementation.

33 AREA 3: PARTNER AND INDUSTRY OBJECTIVES "[eSmart Libraries is] ... an opportunity for libraries to be part of the conversation about the digital world ... [It showcases] that I i b r a r i es a r e j u st as m u eh about d i g it a I Ii t er a cy now as they are about p r i n t Ii t er a cy . "
(Library industry representative, Wave Two) Digital inclusion Wave One and Wave Two evaluation findings identified that library services are among only a few community spaces where community members at different stages of life, and with limited access to technology, can gain both access to and support with technology.
An increased number of staff reported that they provided one-on-one support to library users from Culturally and Linguistically Diverse (CALO) backgrounds (Wave One= 38%; Wave Two= 68%) indicating that libraries were providing a critical avenue for digital inclusion for this group.

Library services across both waves of data reported mixed engagement with young people; this was dependent on library location, access to study space, access to free wi-fi; and/or availability of youth-specific activities.
Library services also reported that older Australians and adu lts with limited digital literacy were benefiting the most from support and access to technology in the library; and discussed many examples of provid ing assistance to these groups.

By ra ising the 'smart, safe and responsible' bar in libraries, eSmart Libraries is contributing to supporting these groups to build their digital literacy.
Findings highlighted that there is more scope to enhance library services' engagement - and thus to impact disadvantaged and marginalised groups - by working w ith the library sector to provide targeted messaging and support to those seeking employment, as well as newly-arrived and CALO community members.

By raising the 'smart, safe and responsible' bar in libraries, eSm art Li b r a r i es i s cont rib u t i n g to su pp or t i n g these groups to build their digital literacy.
"I think a lot of libraries are finding that because people are more mobile, they're given devices, they're not sure how to use them.
A lot of our time now is spent with people, helping them to navigate their way around theonlineworld ... " (Library manager , rural library ser vice, Wave Two) 34 Reaching young people libraries were found to be offering messages consistent to those of eSmart Schools regarding online safety; embedding appropriate policy and procedures to ensure safety; and helping to build a community of understanding and good practice around online safety.
The majority of surveyed young people reported learning about cybersafety at their school (Wave One = 56%; Wave Two = 74%).

Although young people are not the key target group for eSmart, it is important that eSmart messaging in libraries continues to be consistent with that of eSmart Schools.
If the AMF would like eSmart libraries to reach more young people, it could consider a targeted approach focusing on key messages and support for young people who are, in particular: transitioning from education to employment; • • CALO; • outside of mainstream education settings.

35 CASE STUDY: LIBRARY USER #4 (YOUNG PERSON), SMALL REGIONAL LIBRARY SERVICE Kate is a secondary-school student who shelves at her local library after school and is a regular library user.
Her loca I library service has reached eSmart status.

She likes to go to her library to study, use the scanner and photocopier, and use the internet.

Kate has been a library user most of her life, and although she says that she would come anyway, she thinks that the library is more appealing to young people because it has wi-fi and internet access.

She thinks her library is a safe place because of its community feel. "

I feel like I have the right to be safe here" .


Kate thinks young people need to be well-educated before they go anywhere near the internet.

She first learned about the idea of being safe online at school, although she thought that the library had really solidified her learnings.

"[T]he library has helped me more understand it.

n In particular, Kate has learnt more about privacy, ensuring her online profiles have the appropriate privacy settings.
She had picked up brochures in the library that talked through privacy settings. "

[They were] a big eye-opener".

Kate talked about the similarities between what she had learned at school and what she had learned from the library, in particular, about how you can enjoy online technology in a safe way.
36 Cybersafety There was a 7% increase in the number of library staff who reported that eSmart Libraries was making some difference or a significant difference to how they managed cybersafety and cyberbullying in their library (Wave One=== 47%; Wave Two= 54%).

However, the evaluation was not able to determine the prevalence of cybersafety incidents in the library.
Across both waves of data, library staff discussed a number of incidents - such as banking scams, access to pornography and cyberbul!ying - but these appeared to be exceptions rather than the rule.

Staff reported particular concerns about educating library users in how to maintain their privacy online whilst accessing technology in the library.
The AMF's own pre- and post-implementation data seems to suggest there has been a reduction in the number of incidents that negatively affect library staff or users.

There may be scope to further investigate this data with libraries, by undertaking a detailed needs assessment at the beginning of each library service's implementation, so that cybersafety incidents can be tracked over time.
37 AREA 4: PERCEPTIONS Local government " We made a d eci si on to co m m it to [ eSm art Libraries] becausewefeel it is important for us to be a qua I it y service and to support o u r corn mu n i ty."
(Local government representative, metropolitan library service, Wave Two) Local government staff who oversee library services (directors or group managers) have the capacity to play an important role in the implementation of the eSmart Libraries initiative, as they are often the conduit between the library service and the local government executive and elected councillors.
Across both waves of the evaluation, local government representatives affirmed the value of ensuring that libraries are a safe place for library users.

Wave One local government representative discussions focused primarily on the importance of eSmart in assisting library services to demonstrate their risk mitigation strategies in relation to online access in the library.
Wave Two highlighted additional outcomes related to reaching eSmart accreditation; such as how the initiative could: • provide a platform to profile the library's work to senior council management and elected councillors; • enhance libraries' credibility with the community; • improve staff confidence and capacity.
Wave Two interviews with local government representatives also highlighted the importance of continuing to engage with the AMF in the sustaining phase, and included some questioning of how that may look, moving forward.

Opportunities exist for the AMF to capitalise on local government and libraries' support of the initiative by profiling eSmart-accredited library services on relevant media platforms.
The AMF could also seek further opportunities to build and leverage the eSmart Libraries network to promote knowledge-sharing, as well as practical strategies for fostering digital literacy and on line safety between libraries.
Opportunities identified by this evaluation include: • utilising existing library network email lists to establish a community of interest or online biog; • developing a peer mentoring platform and accompanying resources to encourage knowledge-sharing between library services; • providing opportunities for library services to share knowledge and experience of implementing the framework via an annual conference or regular workshop series; • utilising AMF Connect and eSmart Membership to provide ongoing resources and support.
38 The local government manager charged with oversight of the library service suggested that the eSmart library policies could be implemented in other Council services.

The library service has met with staff at a Council youth service facility to discuss whether and how the service could apply the library's eSmart policies.
Library service staff are looking to assist the youth service to implement some eSmart policies; as the service is engaging with young people but does not have in place the same level of cybersafety policy and procedures.

CASE STUDY: LEVERAGING ESMART LIBRARIES POLICIES IN LOCAL GOVERNMENT This library service was a large metropolitan service in a growth area of a major Australian city.

The service reached eSmart status in 2016.


Its municipality is implementing a Smart Cities program, which focuses on liveability, sustainability and economic prosperity, with a strong emphasis on technology.

In a presentation to a Council meeting, the library demonstrated how the eSmart Libraries framework was supporting digital literacy and ensuring responsible, smart, safe access to technology.

This was well received in the context of the Smart Cities program.
"[We talked to] the Councillors about eSmart and what eSmart means and what we have achieved.

That was well received .

So I have to say it was a very proud moment for us."
(Library manager, metropolitan library service) 39 Library community "[l]t was important that we undertook something like [eSmart Libraries] that had a I o t of r i go u r a r o u n d i t [ a n d ] w as a so I i d process that we had confidence in; so that we could get together and go, OK, I think we're doing the best we can in this space ... " (Library manager, regional library service, Wave Two) In Wave Two of data collection, library staff were more likely to recommend eSmart Libraries to another library service (Wave One = 75%; Wave Two = 80%).
The majority of library staff reported that there were benefits to implementing eSmart.

Across both waves of data, staff reported improved staff knowledge and library-user knowledge of how to be smart.

safe and responsible online as the main benefits.

Across the two waves of the evaluation, library services reported greater awareness of the Telstra, AMF and eSmart brands.

In Wave Two, an increased number of library users had heard of eSmart Libraries (Wave One= 23%; Wave Two= 32%).
Among those library users who recognised the eSmart Libraries brand, the.re was a 7% increase in the number that were aware of the Telstra Foundation partnership (Wave One= 41%; Wave Two= 48%).

In Wave Two, close to two thirds of library staff (60%) were aware that the Telstra Foundation was a partner in the delivery of eSmart Libraries, compared to less than half in Wave One (47%).
This suggests that as more libraries enter the sustaining phase, there may be a further positive shift in staffs and users' recognition of these brands.

40 CONCLUSION This evaluation finds that eSmart is making a significant difference to libraries' capacity to develop a systematic and comprehensive approach to the promotion of digital literacy and online safety in Australian libraries.
Comparative findings across the two waves of evaluation show that libraries that have reached eSmart status have more digitally confident management and staff.

The evaluation also highlights that encouragement and support from organisational leadership is key to enabling participating libraries to reach eSmart accreditation.

... eSmart is making a significant difference to libraries' capacity to develop a systematic and corn prehensive approach to the promotion of digital literacy and on Ii ne safety ...
The evaluation shows that, to maximise the impact of the AMF's suite of eSmart offerings, eSmart Libraries would benefit by targeting and supporting specific library user groups who are regular users of technology in the library.

These include older Australians, young people transitioning from education to work, and newly arrived and CALO community members.
Further, to support libraries to maintain the momentum generated by eSmart Libraries, it is important that the AMF identifies opportunities and develops targeted strategies for further strengthening the eSmart community of practice; in ways that deepen knowledge-sharing, peer support, and cross-fertilisation of practical advice, programs and accompanying resources.

Ongoing evaluation of the eSmart Libraries offering will be key to the AMF's capacity to deliver on this.
The eSmart framework improves library users' capacity to engage online in smart, safe and responsible ways in the library by upskilling library staff, improving policies and procedures, and providing access to relevant and quality training and resources.

However, library users do not necessarily recognise the importance of eSmart in supporting libraries to provide spaces where they can go online with confidence.
Cultural change takes time as well as significant background work, and this is not always visible to those who visit library services.

... libraries that have reached eSm art status have more digitally confident management and staff.
The eSmart framework improves library user s' capac i ty to en gage on line in smart , safe and responsible ways in the library ... Whilst library users' awareness of eSmart could be improved, it is more important for eSmart to continue to nurture a community of practice among libraries in Australia, and to nurture the digital skills, competencies and interests of library staff.

As key intermediaries, libraries and their staff are most likely to positively impact library users' online experiences.
41 Finally, the success of the eSmart Libraries framework to date strongly suggests there is significant scope to further build the eSmart Libraries community in Australia , and to translate the framework internationally.

42 REFERENCES Alannah & Madeline Foundatfon (2012).

'AMF briefing document', unpublished.

Australian Government Department of Communications (2013), Referenced in Rolan, G., Denison, T. , Mackenzie, C. , Rolling out Digital Hubs in public libraries: the Mill Park Story.
Paper presented at the VALA 2014 Conference, accessed 30 May 2017 from https://www.vala ,org .au/direct download/vala2014-proceedings/495-vala2014-session-6-roia n-pa per/file Australian Government Office of the Children's eSafety Commissioner (2017).

'eSafe spaces', accessed 22 May 2017 from https://www.esafety.gov au/esafety-information/esafe-spaces Australian Library and Information Association (2016).

The Library and Information Agenda 2016.


Canberra, ACT: Australian Library and Information Association.

Bertot, J. C. (2016).

'Building digitally inclusive communities: the roles of public libraries in digital inclusion and development', paper presented at the Proceedings of the 9th International Conference on Theory and Practice of Electronic Governance, Montevideo, Uruguay.

Bertot, J. C., Real, B.

& Jaeger, P. T. (2016).


'Public libraries building digital Inclusive communities: data and findings from the 2013 Digital Inclusion Survey'.

The Library Quarterly, 86 (3) , 270-289. doi: 10.1086/686674 Bertot, J. C. (2016).

Bu ilding Digitally Inclusive Communities: the Roles of Public Libraries in Digital Inclusion and Development.

Paper presented at the Proceedings of the 9th International Conference on Theory and Practice of Electronic Governance, Montevideo, Uruguay.
Commonwealth of Australia (2016).

'Smart Cities plan', accessed 19 October 2016 from bttos://cities dpmc gov au/smart-cities-plan Fitzgera ld, B., Fletcher, L. , & Kop, T. (2016).

A national approach to digital inclusion and digital literacy: What can this mean for public libraries.

Paper presented at the 18th VALA Biennial Conference and Exhibition, Melbourne.
Accessed 29 May 2016 bttps://www.vala org,au/direct download/vala2016-proceedings/vala2016-Ra.Qers/608-vala2016- session-17-fitzgerald-paper/file GoDigi (2017).

'About GoDigi', accessed 22 May 2017 from https·//www godigi.org au/about Hazlehurst, D. (2015).

'The digital transformation agenda in the 2015-16 federal budget', accessed 17 October 2016 from https://www,dto.gov.au/blog/the-djgital-transformation-agenda-in-the- 2015-16-federal-budget/ 43 Li, J., (2015).
'eSmart Libraries evaluation contextual review: Australian public libraries', unpublished.

Jaeger, P. T., Bertot, J.C. , Thompson, K. M., Katz, S. M. & Decoster, E. J.

(2012).

'The intersection of public policy and public access: digital divides, digital literacy, digital inclusion, and public libraries'.

Public Library Quarterly, 31 (1), 1-20. doi: 10.1080/01616846.2012.654728 McLean, A.

(2016).

'Canberra creates digital transformation agency from OTO'.
Retrieved 17 October 2016 from http://www.zdnet.com/article/canberra creates-dig ital-transformation-agency-from-dto/ Morecroft, R. , Blanchard, M., Third, A, Tracas, G., Li, J., Burns, J., 2015, eSmart Libraries Evaluation: Preliminary Report.

Young and Well CRC, Melbourne.

Morecroft, R., Third, A , Blanchard, M., Tracas, G., Richards, H., Kearney, E. , 2017, eSmart Libraries Evaluation Report: Wave Two Findings, Western Sydney University, Sydney.
Pope, J, Collin , P, Third, A , & Ogun, N (January 2015) eSmart Schools Evaluation 2015.

For Alannah and Madeline Foundation, unpublished.

Telstra (2017).

'About Tech Sawy Seniors', accessed April 2017 from https://www.

telstra.

corn .au/tech-savvy-seniors/about-tech-savvy-sen iors Thomas, J., Barraket, J., Ewing, S., MacDonald, T., Mundell, M. & Tucker, J.

(2016).

'Measuring Australia's digital divide: the Australian Digital Inclusion Index 2016'.
Melbourne: Swinburne University of Technology, for Telstra.

44