ENQA
AGENCYREVIEW

2019

ENQA AGENCY REVIEW:
NATIONAL CENTER FOR EDUCATIONALQUALITY ENHANCEMENT (NCEQE)
MARK FREDERIKS, JANA MOEHREN, MELITA KOVACEVIC, GOHAR HOVHANNISYAN
25 APRIL 2019

...................................................................................................................... 3................................................................................................................................ 4............................... 4
BACKGROUND OF THE REVIEW .............................................................................................................. 4
REVIEW PROCESS ................................................................................................................................ 4........................... 6
HIGHER EDUCATION SYSTEM ................................................................................................................. 6
QUALITY ASSURANCE ........................................................................................................................... 7............................................................................................................................................. 8
NCEQE’S ORGANISATION/STRUCTURE................................................................................................... 8
NCEQE’S FUNCTIONS, ACTIVITIES, PROCEDURES.................................................................................... 10
NCEQE’S FUNDING ........................................................................................................................... 11.............. 13........................................................................ 13
ESG 3.1 ACTIVITIES, POLICY, AND PROCESSES FOR QUALITY ASSURANCE ................................................... 13
ESG 3.2 OFFICIAL STATUS .................................................................................................................. 16
ESG 3.3 INDEPENDENCE .................................................................................................................... 16
ESG 3.4 THEMATIC ANALYSIS ............................................................................................................. 18
ESG 3.5 RESOURCES ......................................................................................................................... 19
ESG 3.6 INTERNAL QUALITY ASSURANCE AND PROFESSIONAL CONDUCT ................................................... 21
ESG 3.7 CYCLICAL EXTERNAL REVIEW OF AGENCIES ................................................................................ 23........................................................................ 23
ESG 2.1 CONSIDERATION OF INTERNAL QUALITY ASSURANCE .................................................................. 23
ESG 2.2 DESIGNING METHODOLOGIES FIT FOR PURPOSE ........................................................................ 25
ESG 2.3 IMPLEMENTING PROCESSES.................................................................................................... 27
ESG 2.4 PEER-REVIEW EXPERTS .......................................................................................................... 29
ESG 2.5 CRITERIA FOR OUTCOMES ...................................................................................................... 31
ESG 2.6 REPORTING ......................................................................................................................... 33
ESG 2.7 COMPLAINTS AND APPEALS.................................................................................................... 34................................................................................................................................. 36............................................................................................. 36.................................................... 36

1/50

........................................................................ 38............................................................................................................................................ 39............................................................................. 39................................................................ 43................................................................................................................... 49................................................................ 50
DOCUMENTS PROVIDED BY NCEQE ..................................................................................................... 50
OTHER SOURCES USED BY THE REVIEW PANEL ........................................................................................ 50

2/50

This reports presents the findings of the ENQA review panel for the Georgian national higher
education quality assurance agency NCEQE (National Center for Educational Quality Enhancement).
The panel reviewed NCEQE against the Standards and Guidelines for Quality Assurance in theEuropean Higher Education Area (ESG) with the purpose of providing information to the ENQA Board
to aid its consideration of whether full membership should be granted and to EQAR to support NCEQE
application to the register.
The judgement by the expert panel against each ESG standard is to provide the ENQA Board and the
EQAR Register Committee with the basis for their respective decisions.
Originally established in 2006, NCEQE has undergone a number of reform processes to arrive at thecurrent external review system, piloted in 2017 and formally operational since 2018. There are twomain external quality assurance activities: mandatory institutional authorization and voluntary
programme accreditation. Programme accreditation is mandatory only for PhD programmes andthose leading to a regulated profession. De facto, nearly all programmes delivered by Georgian higher
education institutions seek accreditation, however, as student funding is only possible in the case ofaccredited programmes.
NCEQE is partially funded by state budget and partially generates its own income through fees for theauthorization and accreditation procedures. In additional to the quality assurance of higher education,other departments of NCEQE carry out functions outside of the scope of ESG, such as the quality
assurance of school and vocational education and the national ENIC-NARIC function.
The ENQA expert panel conducted a review through a desktop study of NCEQE’s self-assessment
report and additional documentation as well as a three-day onsite visit during which discussions witha wide range of stakeholders took place.
NCEQE has the opportunity to provide feedback on possible factual errors of this report.
The expert panel found an agency that was well established nationally and working in close
collaboration with its main stakeholders. A high degree of stakeholder involvement had also
contributed to the development of the new version of standards. These present a significant step inmoving from an input-oriented approach to external QA to an outcome-oriented system which placeda higher emphasis on institutional self-reflection and responsibility – a change that was highly valuedby all discussion partners as well as the expert panel. At the same time, some ways of operating could
be reconsidered in light of new functions, e.g. the way Accreditation and Authorization Councils reach
their decisions and provide guidance on interpretation of the standards to the experts. Additionally,
with regard to the governance system of the agency, the panel noted that this was still indevelopment, both with regard to formal stakeholder representation in the corresponding body andindependence from the Ministry.
The panel found NCEQE to be compliant with the ESG as follows:- Fully compliant for the following ESG: 3.2, 3.5, 3.6, 3.7, 2.1, 2.6- Substantially compliant with the following ESG: 3.1, 3.4, 2.2, 2.3, 2.4, 2.5, 2.7- Partially compliant with the following ESG: 3.3

3/50

This report analyses the compliance of the National Center for Educational Quality Enhancement,
NCEQE, with the Standards and Guidelines for Quality Assurance in the European Higher Education
Area (ESG). It is based on an external review conducted in September – February 2019.

BACKGROUND OF THE REVIEWENQA’s regulations require all member agencies to undergo an external cyclical review, at least once
every five years, in order to verify that they act in substantial compliance with the ESG as adopted at
the Yerevan ministerial conference of the Bologna Process in 2015.
As this is NCEQE’s first external review, the panel is expected to pay particular attention to the policies,
procedures, and criteria in place, being aware that full evidence of concrete results in all areas may
not be available at this stage.
The agency is also applying to be registered on the European Quality Assurance Register for Higher
Education (EQAR).
REVIEW PROCESSThe 2018 external review of NCEQE was conducted in line with the process described in the Guidelines
for ENQA Agency Reviews and in accordance with the timeline set out in the Terms of Reference. The
panel for the external review of NCEQE was appointed by ENQA and composed of the following
members:Mark Frederiks (Chair), International Policy Coordinator, Accreditation Organisation of theNetherlands and Flanders (NVAO), The Netherlands (Quality assurance professional, ENQA
nominee);Jana Moehren (Secretary), Head of Higher Education and Professional Learning, East Kent
College Group, United Kingdom, former Head of International Office of ASIIN, Germany(Quality assurance professional, ENQA nominee);
Melita Kovacevic, Full Professor, Head of Laboratory for Psycholinguistic Research, Universityof Zagreb, Croatia (Academic, EUA nominee);
Gohar Hovhannisyan, Master student at Armenian State University of Economics, Armenia(ESU nominee).

For the ENQA Secretariat, Agnė Grajauskienė coordinated the work of the review panel and attended
the site visit in support of the panel and as an observer for ENQA. The panel would like to thank Ms
Grajauskienė for her invaluable support in organising the process and ensuring the adherence to the
ENQA review principles.
The panel was constituted by ENQA in July 2018. After the reception of the self-assessment report(SAR) in early September, the panel members carried out desktop reviews of the documentation.
During a briefing call in early October 2018, preliminary findings were discussed and request for
further documentation agreed. The process was completed by the site visit and the production of theexternal review report by the panel which was sent to the agency for the correction on factual errors.

4/50

The report is then submitted to the ENQA Board. It is based on the review process, more specifically,
the SAR, agency website, and site visit.
Self-assessment report
The SAR was prepared by a team of NCEQE staff members from different departments beginning inMarch 2018. This working group for the SAR consisted mainly of Head of Departments from different
areas of the agency. A self-evaluation was conducted using the EFQM Excellence Model with theparticipation of a wide range of staff members. The senior management was engaged in the SAR
drafting process on a daily basis while input from different external stakeholders such as HEIs, expert
pool members and members of the Authorization and Accreditation Council was sought throughsurveys and focus group discussions.
The SAR followed the outline suggested by ENQA and provided information on the history and current
activities of the agency, its internal quality assurance and international activities in the first part. Twoof the agencies activities are within the scope of the ESG and thus subject to this review: mandatory
institutional authorization and voluntary programme accreditation. The second part contained theanalysis of the two relevant activities against ESG parts II and III, while the third part contained a SWOT
analysis and a consideration of current and future challenges. Overall, the panel considered the SAR
to be suitable for their review, though additional information was sought prior to and during the visit.
The panel also made use of an agency resource person to clarify procedural and technical questions
about the agency’s relevant activities.
Site visit
The site visit took place in Tbilisi from 29 October – 01 November 2018. The detailed programme canbe found in annex 1.
The visit began with an internal panel meeting to finalize the interview preparation and issues todiscuss. The panel also met with the agency resource person prior to the interviews.
Interviews took place with a wide range of stakeholders, including:











the agency team responsible for preparing the SAR,
the new agency director and the deputy director,
department heads,
staff of the agency,
the Deputy Minister and the former Minister,heads and representatives of quality assurance departments of HEIs,
students,
labour market representatives and partner organisations,expert panel members, andinternational chairs of expert teams.

The majority of interviews took place in English with some translation provided for individual
discussion partners. The interview with the international chairs of expert panels took place via Skype.
A very limited number of participants in different interview rounds cancelled their participation at thelast minute but the panel did not find this to impede the thoroughness of their discussions. Details of
the site visit agenda and the discussion partners can be found in annex 1.
The visit concluded with a final de-briefing. The Secretary drafted the present report, based on the
SAR and findings during the site visit. The report was then circulated to the other panel members for

5/50

feedback and additions. This final version takes into account the comments from NCEQE regarding
factual errors.

HIGHER EDUCATION SYSTEM
The higher education system in Georgia is based on the Law on Higher Education which was adopted
in 2004. In 2005, Georgia joined the Bologna process and started implementing wide-ranging changes
to higher education (HE), including the introduction of a three cycle programme system, ECTS and aquality assurance system.
The law defines three types of higher education institutions (HEIs):

University – all three levels (Bachelor, Master, Doctoral) and research
Teaching University – two levels (Bachelor, Master)
College – one level (Bachelor)

There are 63 authorized HEIs in Georgia, 19 public and 44 private (including 8 orthodox theological
HEIs). At the end of the last academic year (2017/18) approximately 187,000 students were enrolled,
65% of whom in public institutions. The institutions delivered 1,790 degree programmes, 91% of which
are accredited.
A HE Qualifications Framework was approved in 2010 and defines three educational subsystems:
General, Vocation and Higher, each with their own framework. Within the HE framework, seven typesof degree programmes are defined:










Bachelor’s programme (240 ECTS);
Master’s programme (120 ECTS);
Medical education programme (360 ECTS) - one-cycle higher educational programme, Medical
Doctor. Academic degree received after the completion of the educational component with
360 ECTS is equal to academic degree of master’s;
Dental education programme (300 ECTS) - one-cycle higher educational programme, Doctorof Dental Medicine. Academic degree received after the completion of the educational
component with 300 ECTS is equal to academic degree of master’s;
Integrated teacher education programme (300 ECTS) - integrated educational programme,
Master of Education;
Integrated veterinary education programme (300 ECTS) – integrated educational programme,
Master of Veterinary;
Doctoral programme (180 ECTS).

The following picture provides an overview of the education system:

6/50

Diagram 1 (ref. p. 7 SAR)

A review of the National Qualifications Framework has been undertaken since 2014 and relatedchanges in the law on Education Quality Enhancement have been submitted to the Parliament for
approval. The new NQF would include short cycle Bachelor (180 ECTS) and Master (90 ECTS)
programmes aimed at facilitating students’ international mobility.
HEIs in Georgia are autonomous in their academic, economic and administrative activities based on
the law. This also includes freedom of students and staff to carry out academic and scientific work.
Public HEIs are governed by the Academic Council and Senate. Private HEIs are free to determine theirgovernance structure. The involvement of academic staff and students in decision-making and inquality assurance is also stipulated by law, as is the students’ right to quality education and student
self-governance.
QUALITY ASSURANCE
External quality assurance of HE was introduced with the establishment of National Center for
Educational Accreditation (NCEA) in 2006. The first round of reviews, based on quantitative
indicators, was aimed at determining minimum standards for institutions to receive public funding
7/50

and issue qualifications. At the end of this first cycle of institutional accreditations, the number of HEIs
was reduced from about 300 to 67.
With the reform of external quality assurance and the adoption of the Law on Educational Quality
Enhancement in 2010, the system of mandatory institutional authorization and generally voluntary
programme accreditation was introduced. However, for regulated professions and doctoral
programmes, accreditation was set as mandatory. Furthermore, state funding can only be obtained
for accredited programmes, resulting in a majority of programmes being accredited. This approach isstill valid and in use today. At the same time, the external quality assurance shifted from being basedon quantitative to qualitative standards for both authorization and accreditation, though still largely
input based.
A further deep review of standards and procedures took place in 2015-2017, taking into account the2015 version of the ESG and resulting in the system currently in place. The new methodology waspiloted in 2017 and supported by a government project “Study in Georgia”.
The number of external quality assurance processes implemented since 2011 has varied significantlyin the past years. In the year 2018, 27 institutional authorization procedures and 216 programme
accreditation procedures are expected to be completed.

NCEQE was established in 2006 as National Center for Educational Accreditation (NCEA) by orderof the Minister to conduct institutional accreditation. After the first round of the revision of theexternal quality assurance system, the National Center for Educational Quality Enhancement(NCEQE) was established as its legal successor, now under a new law under the Ministry of
Education and Science. The new Law on Educational Quality Enhancement established NCEQE as
independent in its activities and operation, with its structure and responsibilities defined in a
Charter which was adopted in 2010 and last amended in December 2018. The agency’s activities
include external quality assurance of higher education, which is relevant for this revi ew,
authorization of vocational and general education institutions, recognition of qualifications,
governance and management of the NQF, development of sector benchmarks and vocational
education standards and maintenance of a registry of HEIs.
NCEQE’S ORGANISATION/STRUCTURE
The structure of the agency is defined in the Charter. It is depicted in the following organisational
chart.

8/50

Diagram 2 (ref. p 11 SAR)

The Charter defines the Director and since the latest amendment in May 2018 also a new Coordinating
Council as the two governing entities of the agency. At the time of the visit, the Coordinating Council
was not yet established. The panel was provided, however, with a list of proposed members which
had been submitted by the agency to the Minister of Education and Science for approval. The
Coordinating Council shall have between 5 and 13 members representing educational institutions of
all levels, the Ministry, employers and non-governmental and international organisations. Its purpose
is to discuss NCEQE’s strategy and further development of its QA activities.
The decisions about the authorization of higher education institutions and the accreditation of HE
programmes, as described in the next section, are made by the Authorization Council andAccreditation Council, respectively. Both councils’ roles, functions and membership are defined in therespective Charters and the Rule on the selection of Candidates. Currently, both councils consist of 17
members from HEIs, students, employers and NGOs. Its members are appointed by the Prime Minister
upon recommendation of the Ministry of Education, Science, Culture and Sport for one year terms.
Additionally, the Authorization Council has 16 invited members who participate in the discussion anddecision-making only when an HEI is under review which implements medical programmes. The

9/50

Councils are guaranteed independence from HEIs and state organs through the Law of Georgia on
Educational Quality Enhancements.
The Appeal Council has been established in 2016. It consists of eleven members who are nominatedby the Minister of Education, Science, Culture and Sport and appointed by the Prime Minister for aterm of one year.
NCEQE’S FUNCTIONS, ACTIVITIES, PROCEDURES
Two main external quality assurance activities are implemented by NCEQE and are subject to thisreview, authorization of HEIs and accreditation of HE programmes. Both are defined in their respective
Charters which are adopted by the Minister of Education and Science. Both processes include thefollowing steps:-

Submission of self-assessment report (application)
Recognition of institution as applicant
Creation of expert panel
Desk study of application
Site visit
Elaboration of draft report
Submission of feedback by the institution on factual errors in draft report
Elaboration of the final version of the report by the experts and submission to NCEQE
Submission by the HEI of application and expert evaluation report and the argumentative
statement paper to the respect Authorization or Accreditation Council
Council meeting with oral hearing of applicant institution and decision making
Publication of decision and report on NCEQE website

The timeline for both procedures as defined in the charters is 180 days.
Full authorization is granted for six years whereas authorization with the submission of progress report
after one year, authorization with mandatory monitoring visit after two years, authorization without
enrolment of students until fulfilment of recommendations, and denial of authorization are the otherdecision-making options. In all cases, the institutions are expected to submit an interim SER after three
years.
During the decision making for authorization, the Authorization Council also defines the quota ofstudents that can be enrolled during the authorization period. Institutions can also apply for an
extension of the quota during the authorization period. The process for this is stipulated in thecorresponding Charter.
Programme accreditation is granted for four years for new programmes and for seven years in case of
reaccreditation, but only four decision-making options are possible: to grant accreditation, to award
it conditionally for two years with a monitoring visit, to refuse it and to cancel accreditation in case ofexisting programmes. All institutions are also expected to submit an interim SER after three years. In
the case of full accreditation, two options are possible: to grant it fully or to grant it fully, but asking
the HEI to submit a progress report showing how the recommendations were considered.
As of August 2018, the following number of procedures have been implemented both under the oldand new system. During the site visit, the panel understood that all ongoing procedures were expected
to be completed as planned before the end of the year. For 2019, the agency expected
approximatively 20 authorization and 200 accreditation procedures.

10/50

Diagram 3 (ref. p 14 SAR)

The other activities of the agencies – in particular with regard to general up to secondary andvocational education – are not described in any detail in the SAR as they did not form part of thereview.
The agency carries out its processes exclusively within Georgia. With regard to international activities,
the integration of HEIs into the European Higher Education Area is an objective of the agency statedin its Charter. The agency also represents Georgia in the Bologna Follow-Up Group.
A Twinning Project with the support of the European Commission is due to start in January 2019, whichis intended to, among others, support the development of quality assurance and the new NQF.
The agency has collaboration agreements with the Centre for Quality Assessment in Higher Educationof Lithuania and the European Law Faculties Association and has received recognition from the World
Federation for Medical Education (WFME) earlier in 2018.
Other international activities include participation in Tempus projects and cooperation with theEuropean Training Foundation to implement the NQF.
NCEQE’S FUNDING
NCEQE’s budget consists of state funding and own income. The ratio between the two has fluctuatedsomewhat during the past years but in 2017-18 the fees for all of NCEQE’s services have been revisedwith a view of making them suitable to cover all review activities. Accordingly, the part of the budgetthat relies on own income is expected to increase further. Additional income is generated by three
governmental programmes, one of which targets higher education (Internationalization and Quality
Enhancement of Higher Education). State budget income is primarily used to cover generaladministrative costs and developmental activities.

11/50

Diagram 4 (ref. p 29 SAR)

All structural units of the agency participate in budget planning and submit their requests in theplanning phase.
The agency has about 150 staff members, excluding technical staff, with 136 among those on
permanent contracts. The HE Quality Assurance Department, which is responsible for implementing
authorization and accreditation employs 13 full time staff, all of whom have a Master level degree.

12/50

ESG 3.1 ACTIVITIES, POLICY, AND PROCESSES FOR QUALITY ASSURANCE
Standard:
Agencies should undertake external quality assurance activities as defined in Part 2 of the ESG on aregular basis. They should have clear and explicit goals and objectives that are part of their publicly
available mission statement. These should translate into the daily work of the agency. Agencies
should ensure the involvement of stakeholders in their governance and work.
Evidence
External quality assurance of higher education is one of the main functions of the agency andconstitutes the key activity of the corresponding Higher Education Quality Assurance Department. It
is the only agency in Georgia with this remit. Its main objectives, functions and governance are thus
stipulated by the corresponding legal charters. Furthermore, the mission, vision, values and goals of
the agency are published on its website in both Georgian and English language, along with a strategy
for 2016-2020 and annual work plans for its implementation. The agency’s objectives thus are to:-Promote the formation of internal mechanisms of educational quality, implementation of
external mechanisms and their further development through cooperation with educational
institutions and other stakeholders;-

Promote free movement of pupils, vocational students, students, graduates and academic
personnel with a view to ensuring learning, teaching, research and employment in Georgia, as
well as abroad;-

Promote development of quality education culture in educational institutions;-

Promote the integration of Georgian educational institutions into international space andincrease confidence in them.

The two external quality assurance activities of the agency, institutional authorization and programme
accreditation, are implemented in accordance with the standards and procedures defined in theirrespective Charters whereas the organisational objectives, structure and activities of the agency are
set by the Charter establishing the agency.
In both external QA procedures the elements of self-assessment, external expert panel, site visit,
reporting and follow up are present. The current revised system was piloted mainly in 2017, flanked
by extensive stakeholder involvement (SAR, p. 24). Customer satisfaction has been a regular activityof the agency, mainly through surveys of different stakeholder groups, in particular HEIs and students.

13/50

Currently, the agency is in the process of setting up a new Coordinating Council which is stipulated in
the agency’s Charter. This Council is intended to institutionalize stakeholder engagement in theagency governance processes. A process for the selection of Council members, upon suggestion of
NCEQE and appointment by the Ministry of Education and Science, is also defined in the Charter. The
membership of the Coordinating Council calls for 5-13 members to represent heads of educational
institutions (of all levels, i.e. general, vocational and higher), employer and sector representatives,
representatives of international organizations and international specialist as well as representativesof the Ministry, all of whom are appointed for a term of two years. One of the main functions of theCoordinating Council will be to review and amend the action and strategic plans of the agency, toreview and improve the standards and procedures and to discuss the activities and budget of theagency. During the discussions, several stakeholders expressed their expectation that the new
Coordinating Council will originally have a consulting and advisory role but is in the mid-term future
expected to take over a more governing role.
Until the full institution of the Coordinating Council, the Authorization Council, Accreditation Counciland Appeals Council are the formal bodies of NCEQE with responsibility for HE quality assurance. As a
new process, the “Rule on the selection of the candidates for General, Vocational, Higher Education
Institutions Authorization, Educational Programs Accreditation and Appeals Councils’ membership”
will inform the membership of all of these bodies from 2019, i.e. it is now implemented for the firsttime. In line with this rule, a commission established by the Ministry of Education, Science, Culture
and Sport (MESCS) will select the candidates who are then nominated by the Minister for final
appointment by the Prime Minister.
Analysis
External quality assurance of higher education institutions and programmes are the main activities of
the agency with regard to HE. These are carried out on a regular basis and are well recognized andaccepted by the stakeholders. HEIs consider both procedures to be meaningful and to contribute to
the goals of the agency, as well as to support the goals of higher education overall. The panel foundthis confirmed through comments from all stakeholders met during the site visit who expressed a highlevel of detail knowledge as well as satisfaction with the procedures, in particular commending the
new system in place since this year. Furthermore, these goals are made explicit in the formal
documentation and the website. All stakeholders were found to be in general agreement overactivities, policies and processes.
The day to day activities of the agency, more specifically the HE QA Department, evolve solely around
implementing both authorization and accreditation. The review of both procedures since 2017 andthe implementation of the changes made fit well with the development of higher education andinstitutional maturity. The panel found that the change from an input oriented process, consideredmainly bureaucratic, to one focusing on outcomes and enhancement was valued by the stakeholders.
The agency now puts emphasis of its daily activities on the implementation and further refinement of
the revised system. A caveat to this analysis is, however, that the new procedures are only in the earlystages of implementation with the first processes and decisions completed this year. Accordingly, nowhole cycle including all follow-up activities has been completed. In fact, the panel found that thefollow-up and monitoring procedure has not yet been fully developed in all details (cf. ESG 2.3).
While the panel in principal found the different stakeholders to be highly involved in the agency’s
processes and developments, in particular HEIs and students, as well as, to a lesser degree, civilorganisations and employers, the panel noted that the formal representation at committee level was
still lacking. A vacancy in the Accreditation Council for a student representative appeared to take a

14/50

long time to be filled, though it has to be pointed out that student membership is explicitly foreseenin the Charter, so that this is not considered to be a structural but rather an operational issue.
However, the Coordinating Council currently has no student member foreseen in the Charter, so thatstakeholder representation in this regard is lacking. The panel pointed out that this shortcoming would
become even graver when the role of this Council was to be extended in the future. The reasonsprovided for not having student representation were not convincing, i.e. that no process had beenfound so far that would define how students could be nominated. It did not become clear why theprocess for the other members could not be applied, nor why the nomination mechanisms used for
accreditation or authorization could not be used in this instance.
Overall, the panel found that the trust of stakeholders in the system and in the agency, its activities
and decisions was high. This included mechanisms that struck the panel as unusual, for example theshort term of office of only one year for membership in the Authorization and Accreditation Councils
and thus high degree of rotation. However, the reasoning behind this, namely to ensure a high levelof representation of different HEIs and consequently a high level of acceptance of decisions andpromotion of quality assurance in the institutions, appeared sensible to the panel given the national
context and level of development of HEI and autonomy. Nevertheless, this would not preclude aconsideration and possible revision of this practice in due time.
Finally, the panel noted that none of the Councils had any international members. While they did not
find this to impede on the functionality of the system – not least because international experts are
regularly members of panels – this area might again be subject to consideration in the future.
Panel commendations
The progress made by the agency in the past year or so, with the redesign and implementation of the
new system as well as the high degree of stakeholder involvement in these revisions demonstrate agreat commitment to quality enhancement.
Panel recommendations
The involvement of all stakeholder groups in the governance should be improved, in particular since
the Coordinating Council is not yet instituted and will not have a decision-making role. A student
should be imperatively included in the Coordinating Council and efforts should be made to fill current
vacancies.
Panel suggestions for further improvement
While the panel understood the rationale behind the parallel system of authorization andaccreditation (cf. ESG 2.2), it might become sensible to reflect on the balance between effort andcontribution to quality enhancements of the two processes in the future, and when HEIs have gained
more experience in quality assurance.
The panel also encouraged the agency to consider involving international experts in the Authorization
and Accreditation Council, not least as this would contribute to an even greater acceptance of theirdecisions and trust in the processes.
Panel conclusion: substantially compliant

15/50

ESG 3.2 OFFICIAL STATUS
Standard:
Agencies should have an established legal basis and should be formally recognised as quality
assurance agencies by competent public authorities.
Evidence
The status of the agency as sole national body authorized to implement external quality assurance of
higher education is stipulated in the Law on Educational Quality Enhancement, together with theCharter on the establishment of NCEQE as Legal Entity of Public Law. Additionally, the Charter as well
as separate Charters for Authorization and for Accreditation state the role of the agency as well as theprocedures and outcomes of these external quality assurance processes. The agency operates under
the supervision of the Ministry of Education, Science, Sports and Culture whereas authorization or
accreditation decisions are made by the respective Authorization and Accreditation Council, and,according to the SAR approved by the order of the Prime Minister.
Analysis
The panel found that the legal basis in the law and Charters, for the agency as well as its procedures,
was clearly stipulated and transparent for the stakeholders. HEIs demonstrated clarity ofunderstanding of the consequences of the decisions by the agency bodies. The agency and its
decisions are fully recognized by the state bodies and the institutions.
Panel conclusion: fully compliant

ESG 3.3 INDEPENDENCE
Standard:
Agencies should be independent and act autonomously. They should have full responsibility for theiroperations and the outcomes of those operations without third party influence.
Evidence
Organisational independence is regulated in the Law of Georgia on Legal Entities Under Public Law –which establishes that a legal entity of public law is an organisation separated from legislative andstate governing bodies, and which independently, under governmental control, carries out political,
state, social, and educational, cultural and other public activities. NCEQE is thus instituted as such alegal entity. Additionally, the Charters on Authorization and on Accreditation respectively define theprocedures, standards and outcomes of these processes as well as the decision-making and fees. The
Director signs all orders. His role and scope of authority is defined in the above-mentioned law. The
grounds for the dismissal of the director are stipulated in this law as well and related to issues such as
criminal conviction or failing to exercise the powers defined by law in an appropriate manner or for
more than four months.
Charters are orders of the Minister of Education and Science but give operational independence tothe agency. Additional rules and regulations are approved by the director of the agency, for example
regarding internal rules or the selection and definition of the review experts and their role. Membersof the Councils are appointed by the Prime Minister upon submission by the Minister of Education andScience, implementing a new rule for the selection of council members from the end of this year. The
16/50

selection of experts for individual authorization or accreditation procedures is also the sole
responsibility of the agency, as stipulated in the corresponding Rule of Experts’ Activity. Furthermore,ensuring that the experts’ judgements are independent in the sense of independence of outcomes, issupported by a Code of Ethics to which they subscribe. This includes a declaration of no conflict ofinterest. These aspects are also part of the experts’ training (see further ESG 2.4).
With regard to the independence of outcomes, accreditation and authorization decisions lie solely
with the respective Councils. Both Councils make decisions by vote, the results of which are fully
published, i.e. the individual members’ votes are made transparent.
Analysis
The panel took note of the organisational set-up of the agency and the different allocation of powers
and roles of the Ministry, the Director and the Councils. The panel was concerned that the Ministercan dismiss the Director on comparatively vague grounds, i.e. the consideration of what would
constitute an inappropriate manner of exercising the Director’s role. The Director of the agency haschanged rather often in the past years with a new Director just having started a couple of weeks before
the site visit. The panel also considered that the ability of the Ministry to appoint and dismiss Council
members did not support organisational independence. At the same time, the new rules for theselection of Council members do not alleviate this concern as the selection committee itself is set upby the Ministry without any clear definition of how the selection committee members are appointed.
This does therefore not preclude a bias of this committee and entails a risk of making decisions without
objective justification. The panel understood that there was, however, at least some desire within theagency to gain even greater control over the appointment of the Director through the Coordinating
Council or through being able to determine sub laws themselves to increase independence. While
recognizing that the doubts of the panel at least partially related to a new process, i.e. the rule of
selection, the history of many changes in the Directorship, often timed with the appointment of a new
Minister, and the measures to ensure operational independence did not fully convince the panel oforganisational independence. Furthermore, the panel discussed intensively with the different
stakeholders the role of the Minister as member of the Coordinating Council with regard to concernsthat this could impede the independence. Stakeholders from the agency, institutions as well as the
Ministry confirmed both their understanding and expectation that the Minister would merely be one
member among several, and in any case not the Chair of the Council. In this way, they found theMinisters’ membership would strengthen the accountability of the Council, not least as the Ministerwould represent one stakeholder organisation with a policy-making role in educational matters. The
panel understood the reasoning of the arguments brought forward but cautioned how this might
affect independence once the Council has taken up work, and in particular, should it take a more
powerful role within the agency in the future.
With regard to operational independence, the panel was less concerned. The selection andnomination of experts for authorization and accreditation remains fully within the agency and nointerference from either the government or HEIs takes place. At the same time, while the standards
and procedures are stipulated in Ministry approved Charters and while stakeholders are extensively
consulted, their development remains fully within the agency.
Similarly, the panel found that the independence of outcomes was given due to the open voting
practice in the Councils. Even where the actual voting process takes place behind closed doors, thedecision of each council members individually is published in the minutes. The panel acknowledgedthat this was done to ensure full transparency of decision making. The stakeholders felt this wasnecessary due to the small higher education sector where most players know each other. At the same

17/50

time, however, this can cause a risk due to the possible low quorum of only half the members, and noless than 3 present. Furthermore, the panel raised the concern that as voting was fully public, thedecision-making at least incurs the risk that votes are cast to please the applicant institution but also
the appointing body, the Ministry. The panel took into account, however, that all discussion partners
from the Councils felt that their voting was not influenced by either the Ministry or the HEI underreview. The latter was intensively discussed as the HEIs make a representation, oral hearing, during
the Council meeting where decisions are made. The panel considered this practice to be unusual andworthy of further consideration (see ESG 2.2).
Finally, the panel saw at least a risk of favourable voting in line with Ministry expectations due to theshort term of office of only one year in the decision-making bodies.
Panel recommendations
It should be ensured that there is absolute independence from the government in that the Ministercannot dismiss director or council member without serious reasons, the circumstances of whichshould be more transparently defined beyond a mere “inappropriate manner”.
The panel also considered that the agency should take more ownership of how councils as decisionmaking bodies are nominated under the new Rules which give this power to the Ministry.
Panel suggestions for further improvement
The panel understood that a number of the current mechanisms of operation are governed by thenational Administrative Code due to the legal situation of the agency and its bodies. The panel
considered it beneficial, however, for the agency to explore possibilities of how to work outside of theadministrative code to enhance full independence.
Panel conclusion: partially compliant

ESG 3.4 THEMATIC ANALYSIS
Standard:
Agencies should regularly publish reports that describe and analyse the general findings of theirexternal quality assurance activities.
Evidence
Thematic analysis is in development according to the self-evaluation of the agency. Notwithstanding,analysis of the quality assurance processes and results of the previous systems have been carried out
and published in the form of annual reports that have also been discussed with stakeholders. Similarly,
an analysis of the pilot projects of the new system in the previous year is currently being finalized andexpected to be shared and discussed during a conference with stakeholders at the end of this year.
Furthermore, a new Twinning Programme Fiche foresees that a methodology for drafting thematic
analysis and concluding at least one thematic analysis in the higher education sector is one indicatorof achievement (component 1).
Analysis
The panel considered that the analysis made in the previous annual reports went some way towards
a thematic analysis. The aim of showing developments, trends and areas of good practice was also

18/50

supported by the annual conference. While not primarily produced with the intention of a thematic
analysis in the sense of the ESG, the panel found that previous annual reports included analyticalsections of the results of the procedures. This practice had been implemented in previous years under
the old system and was ongoing for the new system at the time of the report. The panel noted
positively, that the analysis reports were always discussed with the stakeholders, thereby contributing
to a sector-wide analysis and reflection.
It was, however, brought to attention that the agency staff feels the lack of applying analytical skills
during the work. In this regard, the panel also noted that the data collection was not yet systematic inthat there was no clear use of the big amounts of data and information collected from theauthorization and accreditation processes to feed into a consistent analysis of QA outcomes.
Overall, the panel therefore supported the plan to develop a methodology for thematic analysis in theupcoming Twinning Project, which they understood should start in 2019 for a period of two years. In
addition, the panel acknowledged the finalisation of the analysis of the pilot projects which will bepresented at a conference during the end of the year, and appreciated that both the pilot projects
itself and the analysis required major efforts and resources.
Panel recommendations
The agency should make use of the Twinning project to establish a sustainable methodology for
implementing systematic analysis, also beyond the lifetime of the project.
Panel conclusion: substantially compliant

ESG 3.5 RESOURCES
Standard:
Agencies should have adequate and appropriate resources, both human and financial, to carry out
their work.
Evidence
The agency’s financial resources stem from two main sources of income, state budget and own income
generated through fees. In the past three years, each have made up roughly half of the income. The
budgetary, i.e. state income, is mainly used for general administrative costs and activities. Additional
state income from a government programme “Higher Education Quality Enhancement andInternationalization” is also of relevance as the costs for capacity building, the annual conference aswell as the costs for international experts are covered by this programme.
The fees for authorization and accreditation procedures constitute the main source of own income.
They have been revised in 2017/18 in consultation with the HEIs with a view to making themreasonable for the institutions and increasing the own income of the agency at the same time. During
the discussions, all stakeholders expressed their satisfaction with the fee structure.
With regard to personnel resources, the agency has 149 staff members, excluding technical staff, ofwhom 136 are on permanent contracts. All staff have at least a degree, with more than 70% with a
Master degree. Most staff have been employed for more than five years.
The Higher Education Quality Assurance Department, which is responsible for managing theauthorization and accreditation procedures, currently has 13 full-time staff members. There are also

19/50

immediate plans for recruiting more staff to the department by the beginning of 2019 for which thebudget is already planned. Based on the expected increase in the number of procedures in the nextyear, the accreditation department alone is budgeted to have 12 staff members. Other departments
include the Strategic Development and International Relations Department, Internal Audit
Department as well as HR and Financial Divisions.
Staff development is implemented based on the needs identified by the different departments. In thecurrent year, training with particular relevance to the accreditation and authorization staff wasdelivered in conflict management. A specific budget line is allocated to staff development.
Material resources include the agency offices which are their own property including offices andmeeting rooms as well as the infrastructure with computers and other equipment.
Analysis
The panel considered the resources of the agency to be suitable to implement effective and efficientquality assurance processes. The income of the agency was built on a suitable model of Ministerial
and own income with a balance shifting towards increased own income. This supported the
independence from the Ministry, and the panel gained the impression that all stakeholders considered
the fees to be paid for the procedures to be a suitable means of securing agency income. In particular,
the new system of fees based on the number of programmes, breadth of the programmes and numberof experts in the review team was an improvement of the old lump sum based system which didn’t
cover the costs of all procedures. Furthermore, the panel noted that an underspent from the previous
and current year had occurred demonstrating a sustainable use of resources. The agency also providedplans for spending the surplus, namely by financing services which are offered free of charge, such asrecognition of student mobility and development oriented trainings. For the upcoming year, six
training sessions are also planned for authorization and accreditation which are free to theparticipants thanks to the income generated. The panel found this approach to be laudable.
While the panel noted that the agency determined their own budget and the usage of its income, thepanel took note that the approval processes for the budget through the Ministry was rather longwinded. This meant that the agency has to submit their budget, including spending proposals in May
but it is only approved in December. This did not impede the functioning of the agency activities,
however, not least as the agency can shift up to 10% among the different budget lines without
additional ministerial approval.
With regard to personnel resources, the number of staff in the HE Quality Assurance Department was
low compared to the overall staff and the high workload. In particular, the panel noted that there
were peaks of activity for the department when many procedures had to be implemented at the sametime. The remedial actions put in place for these cases were, however, found to be adequate. Morespecifically, staff from one unit would be called to support another unit with a particular highworkload. The additional costs for this are planned for in the budget for the next year, thoughreallocation of resources is challenging when the Ministry has to authorize these. The panel also notedthat new staff would be hired shortly to strengthen the department.
Panel suggestions for further improvement
It would be beneficial to gain a bit more than 10% flexibility on how the budget is distributed once
approved by the Ministry.
Panel conclusion: fully compliant

20/50

ESG 3.6 INTERNAL QUALITY ASSURANCE AND PROFESSIONAL CONDUCTStandard:
Agencies should have in place processes for internal quality assurance related to defining, assuring
and enhancing the quality and integrity of their activities.
Evidence
Since 2015, the internal quality assurance of the agency is informed by the EFQM model as stipulated
in the “Quality Assurance Mechanisms of NCEQE’s Activity”, an annex of the internal rules of NCEQE
which are approved by the Director. The EFQM Excellence Model tools are used for the planning,
implementation, assessment and enhancement of the agencies activities. In line with the model, an
institutional self-assessment is carried out every two years. The last external revision led to the status“Committed to Excellence” and the action plan for 2018-19 calls for the further implementation of the
EFQM tools.
The implementation of the strategy and of the work plans of the structural units is overseen by theStrategic Development and International Relations Department. Annual reports summarize the main
activities and performance of the units and are published, in Georgian language, on the website.
Weekly staff meetings of the agency leadership with the heads of all units are part of the day-to-day
quality assurance. The communication from these meetings are cascaded to all staff through weekly
meetings within the units.
Stakeholder involvement and the collection of stakeholder feedback are an important part of theinternal quality assurance and are mainly implemented through surveys. Internally satisfaction
surveys of staff are conducted while externally customer satisfaction surveys are commissioned. The
results are then analysed and discussed and put into action where appropriate.
With regard to quality assurance specific to the authorization and accreditation procedures, after eachsite visit, the institutions are asked for feedback about the panel. Additionally, the panel members are
asked for their feedback and the chair provides feedback on the panel members. Further feedback iscollected from events organised by NCEQE and from complaints or applications.
The Internal Audit Department monitors the implementation of the internal quality assurance. Since
2015 it carries out financial, operational and compliance audits to ensure that the NCEQE and all of its
structural units comply with their own regulations and charters as well as other legislativerequirements. The Department also monitors whether recommendations from audits are
implemented. To this extent, the department fulfils the function of internal quality assurer also forthe authorization and accreditation departments who themselves also analyse the feedback fromtheir stakeholders to feed into revision of procedures and criteria. Accountability to its stakeholders,in particular the government, HEIs and experts is achieved through information and publication of
information on the agency’s website. Additionally, formal and informal discussions andcommunication between the agency and the national stakeholders take place all the time, specificallywith the Ministry and institutions’ representatives.
NCEQE’s staff undergoes performance reviews twice per year and development needs may be theresult of these assessments, which would in turn lead to personal development and training plans.
There is also a code of conduct in place for staff members.

21/50

The competency and conduct of the experts is ensured on the one hand through training and on theother hand through a Code of Ethics for experts. This includes the prevention of any conflict of ethics.
Furthermore, the Public Administration Act applies to all staff, council members and experts andgenerally regulates the avoidance of conflict of interest.
The principles of equality are considered to be part of the commitment to quality and are inherent toall policies and working practices, for example with regard to the treatment of applicant institutions.
AnalysisInternal quality assurance was found to be supported by the EFQM model tools. The panel noted that
evaluations and surveys are a key part of the internal quality assurance system in addition to otheractivities such as round tables and meetings. Overall, the internal quality assurance system was foundto be very comprehensive. While many measures of improvement based on stakeholder feedback are
implemented by the Department of HE Quality Assurance itself, the Internal Audit Department
supported the systematic implementation of their process and improvements as a whole. Morespecifically with regard to authorization and accreditation, the feedback from and communication
with stakeholders was found to be extensive. NCEQE is thus well placed to make use of the large
amounts of data it obtains through surveys and meetings. In all interviews, but particularly with HEI
representatives and staff, the panel gained the impression that the agency takes the continuous
development and improvement of its external quality assurance procedures very seriously. This is notleast evidenced by the fact that the current version of standards and procedures was developed basedon experiences and identified areas for improvement from earlier versions, and with wide-ranging
stakeholder input to guarantee acceptance by them. The consultations around the preparation of theexternal review have also supported the elaboration of the annex to the Charter which for the firsttime systematizes the elements of the internal quality assurance.
Furthermore, the panel commended that a so-called “quality school” will be implemented from next
year with the aim of further improving the quality assurance competence of both staff and also
external experts in the institutions’ QA departments. This would allow a further enhancement ofexpertise and thereby benefit both external and internal quality management.
In terms of the professional conduct of staff and guards against discrimination of any kind, the panel
learned that this was governed by the General Administrative Code of Georgia to which the agency
and its staff had to apply. Additionally, the internal regulations stipulate how the agency would handle
any cases of complaints, appeals or misconduct. These are available in Georgian on the website. The
role of the Internal Audit Department in dealing with such cases is also stipulated in the agency
Charter. The panel was pleased to hear that no cases have occurred yet. The panel had no indications
that discrimination or intolerance were a risk. They considered, however, that the professional
behaviour of agency staff would go beyond the disciplinary procedures to which the agency referredin this context. Despite the facts that guards against the risk of discrimination or intolerance are partof general national codes, the panel was not convinced that the agency and staff members were fully
aware of their importance as some staff struggled when discussing these issues with the panel.
Further understanding of this area might thus be helpful in particular where the specifics of external
quality assurance make a deviation from the general administrative code sensible.
The panel also established that NCEQE currently does not work with any subcontractors in carryingout activities in the scope of the ESG.

22/50

Panel commendations
The panel commended NCEQE on the use of the EFQM Excellence model as well as on the thoroughsystem of evaluations in place.
Panel suggestions for further improvement
NCEQE may wish to expand and revisit their code of conduct and the internal procedures withregard to ensuring that guards against intolerance and discrimination are also formally in place.
Panel conclusion: fully compliant

ESG 3.7 CYCLICAL EXTERNAL REVIEW OF AGENCIESStandard:
Agencies should undergo an external review at least once every five years in order to demonstrate
their compliance with the ESG.
Evidence
This is the first formal external review of NCEQE against the ESG 2015. Previous external reviews
against the ESG with a purpose of development have been carried out in the frame of EU-financed
projects and have contributed to the establishment of the current authorization and accreditation
standards.
Analysis
While this was the first formal review against the ESG for the purpose of full ENQA membership andinclusion in the EQAR, the panel acknowledged positively that informal reviews had been
implemented previously and had been used to substantiate the further development of NCEQE. The
panel appreciated that the agency plans to undergo cyclical regular ESG reviews, as is required for
ENQA membership and EQAR listing.
Panel conclusion: fully compliant

ESG 2.1 CONSIDERATION OF INTERNAL QUALITY ASSURANCE
Standard:
External quality assurance should address the effectiveness of the internal quality assurance
processes described in Part 1 of the ESG.
Evidence
The ESG standards of part 1 are implemented into the standards for authorisation and accreditation
as stipulated by the respective Charters. The agency provided the following alignment of the ESG part
1 with both procedures.

23/50

Diagram 5 (ref. p 36/37 SAR)
24/50

Each of the standards is detailed in terms of evaluation criteria, and evidence and indicators for eachcomponent in the respective Charter and the corresponding templates for the self-assessment and forthe experts’ report. The main change identified for the new sets of standards is the encouragement
to institutions to engage all stakeholders in the process and strive for quality enhancement rather
than merely formal compliance with standards.
While there is some overlap in the standards between authorization and accreditation, this is intended
and substantiated through the different foci of both procedures, namely the institution as a wholeand its processes in authorization and individual programmes in accreditation.
Analysis
The panel reviewed the alignment of the authorization and accreditation standards with part 1 of the
ESG as well as how this translated into actual reports. All ESG standards of part 1 are represented inboth of NCEQE’s procedures, albeit adapted to the nature of the process. Thus the emphasis ofauthorization as mandatory external quality assurance is different from that of programme
accreditation which is only mandatory for regulated and Doctoral programmes. Nevertheless, thepanel found that both fully incorporate ESG part 1 at the moment.
The panel did point out however, that the standards for programme accreditation, which is explicitly
required for Doctoral programmes did not contain any specific standards on research. While thequestion of research capabilities and output is already part of mandatory institutional authorization,
which would have taken place before any programme accreditation, it might be worthwhile reconsidering how this important aspect is treated in terms of Doctoral programme accreditation. The
panel welcomed that deliberations had been started with the stakeholders to draft separate standards
for PhD programmes.
The panel confirmed how the ESG part 1 are dealt with in practice by analysing a sample of reports
obtained from the agency’s website. These were found to satisfactorily address the standards of ESG
part 1 through analysis of the corresponding agency standards.
Panel suggestions for further improvement
The panel would like to encourage the agency to continue exploring the opportunities for how to deal
with research elements in programme accreditation, in particular accreditation for PhD programmes.
Panel conclusion: fully compliant

ESG 2.2 DESIGNING METHODOLOGIES FIT FOR PURPOSE
Standard:
External quality assurance should be defined and designed specifically to ensure its fitness to achieve
the aims and objectives set for it, while taking into account relevant regulations. Stakeholders should
be involved in its design and continuous improvement.
Evidence
NCEQE implements two forms of external quality assurance, as mentioned above: mandatory
institutional authorization and voluntary programme accreditation (mandatory for Doctoral
programmes and regulated professions). While both procedures have been in place in principle since
the foundation of the agency, they have been re-designed and improved over the years. Stakeholders,25/50

in particular the Ministry and HEIs, are always closely involved in those reforms. In the latest reform,
four main phases were implemented: Firstly, working groups made up of staff members, Ministry,
public bodies and HEI representatives drafted the first version. Secondly, stakeholder feedback on thedrafts was sought from all stakeholders, including national and international experts. Thirdly, the draft
documents and feedback were discussed at a national conference and further feedback frominternational experts was incorporated into the final drafts. These were then, fourthly, tested in pilot
procedures.
The methodologies are also described in a guidebook for authorization, with a version for
accreditation in progress. The main focus on the new methodologies for both processes is toencourage institutions to develop a quality culture and to strengthen self-assessment and reflection
capacities by having moved from an input-oriented to an output-oriented approach.
Analysis
The involvement of stakeholders in the review and redesign of external quality assurance
methodologies was found to be a key strength of NCEQE. During all discussions, the interviewees
demonstrated a high level of satisfaction with the work of NCEQE in general and the revised
procedures in particular which constituted a significant step forward in effective external quality
assurance. The panel found substantive agreement among all stakeholders that the methodologies
were fit for purpose. It was also highlighted that they had a positive impact on the actual quality ofhigher education in the institutions. Additionally, the involvement of the labour market and employers
in external quality assurance was strengthened as NCEQE had found ways of involving them in thereview processes despite a low level of organisation among employers and employer organisations
which are not very active.
The panel also considered and discussed in detail with the stakeholders the effects of the current twotier system and the possible related burden versus benefit for the institutions. This is because
programme accreditation, though formally only mandatory for a limited number of programmes, is defacto implemented for nearly all programmes due to the student funding which can only be obtained
for accredited programmes. Nevertheless, the panel found that a large support exists for having both
systems in place as they have different focus areas and thereby were considered to help institutions
to develop sound internal quality management systems. Once these will be in place in the future, areconsideration of the current approaches will become sensible.
One topic of concern for the panel was the fact that as part of authorization decisions, a decision ismade for student quota, i.e. how many students the HEI will be allowed to enrol in its programmes.
The panel understood that in this regard, too, a revised system had been put in place in the last reform.
Institutions are now asked to make reasonable calculations and proposals themselves which are thenput forward to the expert panel which has to determine whether the institutions proposal was
reasonable. The final decision is then taken by the Authorization Council, together with the decisionon all other standards. While the panel acknowledged that this was a step forward, they wereconcerned by the high burden this analysis and decision placed on the expert panels with regard tocomplex calculations and estimations. Usually, such an issue would be outside of quality assurance
processes for institutional quality based on ESG part 1. While there were some indications on how to
calculate student quota in the guidelines for HEIs, the panel considered the corresponding standard
and guidance for experts to be not sufficiently helpful to enable them to make such a decision.
The panel understood that seven joint programmes had been reviewed by the agency so far, basedon NCEQE’s own standards. The panel also learned that the development of joint programmes was

26/50

part of a new framework that would enable institutions to implement further joint programmes. The
panel also noted that the use of the European Approach for Quality Assurance of Joint Programmes
was not embedded in the processes. While the panel understood that a Director’s Order (N74 of
02.03.2017) was in use for carrying out joint programmes, the panel was not made aware of any usageof the European Approach. Similarly, with regard to international cooperation among agencies, NCEQE
stated that they would recognise automatically decisions by other agencies listed on EQAR; and thatin other cases they would compare the standards and procedures used. However, this process has
been roughly defined in terms of ENQA members in European Union countries and agencies from theUnited States whose decisions will be accepted. For other agencies, the processes must be compatible,
though it is not defined how such a compatibility would be established. There is, furthermore, no
specific detail on how the necessary follow-up of decisions in such cases would take place, e.g. by theforeign agency or NCEQE.
Panel commendations
The panel commends the close involvement of stakeholders in the revision and redesign of theagencies’ procedures.
Panel recommendations
It is recommended to use the European Approach for Quality Assurance of Joint Programmes.
It is recommended to detail and specify how recognition of decisions or reviews by international
agencies would function, for example with regard to establishing compatibility and implementing
follow-up procedures.
NCEQE is recommended to provide more guidance to experts and the Authorization Council on how
to analyse the standard dealing with determining students’ quota.
Panel conclusion: substantially compliant

ESG 2.3 IMPLEMENTING PROCESSES
Standard:
External quality assurance processes should be reliable, useful, pre-defined, implemented
consistently and published. They include:-

a self-assessment or equivalent-

an external assessment normally including a site visit-

a report resulting from the external assessment-

a consistent follow-up

Evidence
Both institutional authorisation and programme accreditation include the following steps which aredefined in the respective Charters and are published on the agency’s website.
Self-Evaluation is the first step and is conducted by the HEIs based on the respective templateapproved by the NCEQE Director. The templates provide indication of possible evidence andencourage the institution to reflect on their performance.
27/50

The authorization or accreditation is then formally initiated by the submission of the self-evaluation
report.
The external assessment begins with setting up the panel and the determination of the timeline forthe process which is set in an order of the Director. The assessment itself is implemented through adesktop review of the self-assessment report and appendices by each expert and a subsequent onsite
visit which lasts 3-5 days, depending on the size of the HEI, or 1 day in case of accreditation. During
the site visit, interviews are carried out with different stakeholder groups of the institution.
The report of the findings from the desktop study and onsite visit, analysed against the respective
authorization or accreditation standards, is produced after the visit. A draft version is provided to the
HEI for feedback on possible factual errors. Subsequently the final version is the basis of the decisionof the respective Council.
A follow-up for all HEIs takes place three years after the decision when they have to submit an interim
institutional or programme self-assessment report. Further mandatory follow-up depends on the levelof compliance with the standards, i.e. in case of authorization for six years and of conditional
accreditation this can be a progress report after one year and monitoring after two years. In case of
authorization, HEIs can also be required limit enrolment of students from 1 to 3 years. Additionally, acase-based monitoring can also take place when a concern is raised to the agency by a student or astaff member of an institution. In these cases, the agency, by order of the Director, will set up an
expert group to investigate the concern.
Analysis
The panel found the principal steps of the authorization and accreditation processes to be transparent
and systematic. The newly developed guidelines and templates were confirmed by the stakeholders
to be useful and support a more effective implementation. In particular, the timelines are now morereasonable and vastly adhered to, which is also checked by the internal audit department. This
includes, for example, that the expert panel receives the self-assessment report and additional
documentation much earlier before the site visit than previously. The site visit was also found to be ofadequate length, not least due to the involvement of international experts and related necessarytranslations. While translations of documents in authorization processes provided a challenge for theagency and the institutions, specifically with regard to timely delivery, all agreed that the benefits ofexternal experts (cf. standard 2.4) outweighed the additional work attached to it.
However, as the new procedures had not yet completed a full circle beyond the authorization or
accreditation decision, the follow-up procedure was not yet developed in all of its details. The experts
questioned the purpose of the intended mandatory interim self-assessment after three years,
especially in relation to one-year progress reports or two-year monitoring. However, the national
stakeholders confirmed that previously a report had to be submitted every year which constituted arather formal exercise as little feedback occurred. The new approach was expected to support theinstitutional development by keeping up momentum in the development of quality assurance inbetween formal reviews. While the experts concurred with this purpose, they were not convinced thatthe effects on the workload had been fully anticipated and whether the human resources would be
adequately utilized. Furthermore, the monitoring process remained unclear, as it appeared to be
initiated on a case-based basis. It was not evident to which extent this type of monitoring would
complement or supplement the mandatory follow-up element of authorization or accreditation. Whilerequests and inquiries could come from different sources and through different channels, for example

28/50

through contact by students with the agency, the terms, conditions and limits of this case-based
monitoring did not appear to be fully defined and thus were not entirely transparent.
Panel recommendations
NCEQE has to define the details of the different follow-up procedures.
Panel suggestions for further improvement
The panel encourages the agency to reconsider whether they will need such close monitoring of theinstitutions in due time in order to place more emphasis on institutional autonomy and responsibility.
Panel conclusion: substantially compliant

ESG 2.4 PEER-REVIEW EXPERTS
Standard:
External quality assurance should be carried out by groups of external experts that include (a)student member(s).
Evidence
External experts form the review panels in both authorization and accreditation procedures as definedin the respective Charters. Currently, the registry of experts has 350 national experts for accreditationand 200 international experts for accreditation, 173 national authorization experts and 42
international authorization experts.
Experts are selected through a procedure defined in the Rule of Experts’ Activity and include anapplication and selection interview. The Rule also sets out the required qualifications. NCEQE launcheda public call for experts’ applications during the reform of the system in 2016 and also considerednominations from institutions.
For each procedure, the selection of the experts follows the Rule of Experts where the composition of
the panel is defined. According to the Charter, the panel in authorization procedures is composed of
5-8 experts, depending on the size of the institution. The chair is always an international expert withstudents and employer representatives other mandatory members. The rest of the panel is made upof academic and administrative staff from HEIs. In programme accreditation procedures, the panel iscomposed of field specific experts including a student and usually an employer representative. In thecase of medical programmes, one of the experts is international. Furthermore, in the case of all
programmes delivered in English language, doctoral programmes and regulated fields, aninternational expert participates.
For case-based monitoring procedures depending on the matter discussed the expert panel might becomposed of at least two experts that might not include student and employer representatives.
All experts have to adhere to the Code of Ethics which includes issues like no-conflict-of-interest, e.g.
by not having had a working relation with the institution under review in the past two years. Experts
are trained in regular trainings, some of which have been financially supported by the National
Erasmus+ Office. It is mandatory for all experts to participate in the training, though international
experts are exempt as the training is implemented in Georgia. They are nevertheless provided withwritten information about the Georgian education system and the NCEQE approach to external quality

29/50

assurance. At the beginning of each site visit, a preparatory meeting is held among the expert groupwhere they are again provided with information and guidelines and review the standards andprocedures. Finally, the Rule also stipulates that the institutions provide feedback after the procedure
as does the chair of the panel on the other expert panel members. The survey results are used to give
feedback to the individual experts and also to plan future capacity building.
Analysis
The role of the experts and the composition of the authorization and accreditation expert groups areclearly set out in the agency’s regulations. The panel considered that training of the experts was
overall extensive and that usually all experts undergo training. However, there didn’t appear to beclear systematic plan as training on additional topics was offered in a somewhat irregular manner, i.e.
when external or project funds were available. This was in line with feedback from experts that thetraining provided was a very good start but that it could be done in a more consistent manner. The
panel did explicitly support the idea of ensuring that more expertise nationally was continued to be
developed. Nevertheless, the inclusion of international experts was mentioned as particularly
beneficial by several interviewees in that they bring an external, unbiased perspective. The panel also
acknowledged that English language skills were required for the national experts to facilitatecommunication among the team.
The international experts do not participate in training sessions per se but are provided with a briefing
pack which describes the Georgian HE system. Despite the experts confirming that this was a goodbase for understanding the Georgian system and particularities, in addition to the briefing meeting at
the beginning of the site visit, they agreed that a bit more information would be helpful beforehand.
The panel also noted that local experts would find it helpful to have more time to work withinternational experts before the site visit. To this regard, the considerations of NCEQE to develop an
online training are welcomed.
During the discussion the panel learned that some experts participate in a rather high number ofprocedures, sometimes both authorization and accreditation. While this was occasionally necessary
for very specific subject areas, experts were also used in more generic accreditations or authorization
procedures where the reasons did not become evident. While the panel recognized that the selectionof experts also depends on their availability, the panel felt that the agency would benefit fromselecting experts more carefully and to continue activities to diversify the expert base. While the panel
acknowledged that establishing a large expert pool was not always straightforward, they pointed outthat the system overall would benefit from building more local expertise, thereby also addressing theissue often mentioned, namely that of a small HE community where many collaborations exist.
As opposed to authorization procedures, the expert panels in accreditation procedures did not always
include an international expert. This would only be the case if the programme was taught in English.
Furthermore, the panel learned during the discussions that also an employer representative was notalways a panel member – while students and HEI representatives were. This struck the review panel
as somewhat odd and not in line with the purpose as programme accreditation was only mandatory
for programmes in regulated professions (apart from Doctoral programmes), i.e. those were theprofessional, employer perspective would be particularly important.
The panel have also learned that for follow-up and case-based monitoring procedures combination of
the expert group is not clearly defined.

30/50

Panel recommendations
The approach to the training of the experts should be revised, with a particular focus on more detailed,informative briefing for international experts, also available at a longer time ahead of the onsite visit,
as well as on more systematic joint training of national and international experts.
Panel suggestions for further improvement
The option of using online resources for expert training and collaboration among experts before thevisit should be further developed.
NCEQE should consider how often individual experts are part of procedures.
NCEQE should ensure that employer representatives are consistently members of accreditation expert
groups.
Panel conclusion: substantially compliant

ESG 2.5 CRITERIA FOR OUTCOMES
Standard:
Any outcomes or judgements made as the result of external quality assurance should be based on
explicit and published criteria that are applied consistently, irrespective of whether the processleads to a formal decision.
Evidence
Both the institutional authorization and programme accreditation standards are the basis of therespective analysis and scrutiny of expert panels and the final decision by the respected Authorization
Council or Accreditation Council. The standards are defined and published in the corresponding
Charters, with a guidebook and templates available for the self-assessment report and the expert
report to support consistent analysis of the institutions.
The expert panel’s report contains a judgement on the level of compliance for each standard and therationale for defining each of these levels is detailed in the Charter of Authorization and Charter ofAccreditation respectively. The evidence used by the panel is also listed in their report. Experts’consistent application of standards is supported by their training, as well as by the templates andguidebook which provide guidance on which evidence to consider. Furthermore, a staff member of
the agency who accompanies the expert panels, provides support in interpreting standards and thecorresponding expectations.
The respective Councils receive the full application from each institution or programme, together withthe experts’ report for their decision-making. The possible outcomes of this process are also definedin the Charter. During the Council meeting, representatives of the HEI or programme under review
make a representation to the Council and give arguments to their position. The corresponding expert
panel members are also invited to participate, though they cannot always all be present and theinternational experts would usually join via videoconference.
Voting then takes place without the institution present. Detailed records of the votes, of eachindividual councilmember on each standard are published on the NCEQE website in the minutes of
the Council meeting.

31/50

Analysis
The Authorization and Accreditation standards, against which the decisions are made, are clear andtransparently published on the website. They form the basis for the outcomes, i.e. the analysis by theexpert panels and the decisions by the respective Councils.
The panel found the experts’ reports (see further ESG 2.6) to be a consistent representation of theexperts’ analysis of the different institutions and programmes. In this regard, the panel considered it
helpful that the agency staff member checked all reports for completeness, comprehensiveness andconsistency, in particular with regard to the analysis and the assurance that all recommendations are
based on evidence and consistent evaluation of facts.
The members of the Authorization and of the Accreditation Council are appointed for one year. They
can be reappointed, in principle for an unlimited number of years. The panel considered this shortformal term of office to be counter-productive to ensuring a consistent – over the years – decisionmaking, though they did acknowledge that many Council members stayed in their role for a few years,
being reappointed for several consecutive one year terms. The panel did not encounter any instancesof issues solely attributed to this fact. They also understood that this arrangement had been
introduced partly to give as many institutions as possible the chance to be involved at this level ofauthorization or accreditation. Nevertheless, the panel did not see evidence that a significant
institutional memory of decision-making, i.e. of a joint interpretation and understanding of thestandards and requirements was achieved, which carried with it a risk regarding this aspect of thestandards and procedures. The NCEQE staff members who coordinated the review procedures
remained in their passive role as providing advice and information only when asked. While this had
not posed any problem so far, the panel also pointed out the risk of not limiting the number of termsof office for an individual. Some of the discussion partners supported the idea of longer terms of office,such as 2 or 3 years, while others had a preference for the current system. In any case, the panel foundthat some discussions about this question were ongoing.
The Authorization and Accreditation Council received and reviewed the whole self-assessment anddocumentation from the institutions in addition to the experts report. The panel understood that thelevel of detail to which the Council members analysed the documentation depended to a large extent
to the representation the HEI made before them and how much they agreed with the panel findings.
Both elements could and do sometimes lead to the Council members almost taking over the role ofexperts and, in turn, making new judgements. While it is reasonable that decision-making bodies
sometimes amend the experts’ proposals, not least to ensure consistency of decisions, the systemused by NCEQE could lead to the opposite. It should be noted that the discussion partners voiced very
little concerns over this system, however, though some HEIs were unhappy and uncomprehending
when the Council changed the experts’ proposals for judgement.
The voting in the Councils takes place behind closed doors. However, the details of vote, i.e. thedistribution of votes, in difficult cases per standard, is published in the minutes. While the panel had
understood from different meetings during the site visit that each members vote was also publishedso that everyone can see which Council member voted in which way, the agency confirmed that thiswas not the case. The panel understood that this was practiced to achieve a high level of transparency.
But at the same time, the panel could not preclude that this could not lead to favourable voting,though no discussion partner conceded this risk.

32/50

Panel recommendations
NCEQE should assess whether the terms of office, working methodology and voting methodology of
the Authorization and the Accreditation Councils contribute to systematically ensuring consistency.
Panel conclusion: substantially compliant

ESG 2.6 REPORTING
Standard:
Full reports by the experts should be published, clear and accessible to the academic community,
external partners and other interested individuals. If the agency takes any formal decision based on
the reports, the decision should be published together with the report.
Evidence
NCEQE provides templates to the expert panels for both authorization and accreditation procedures
which include an overview and context of the institution or programme, information about the site
visit and the panel members, the quality of the self-evaluation report and the compliance with eachstandard. The latter is divided into evidence and indicators, an analysis of those as well asrecommendations (necessary to meet a standard), non-binding suggestions for further improvement,
best practices and a judgment on the level of compliance. The reports are usually written by different
panel members, upon discretion of the chair who determines the internal working method.
Accordingly, the chair decides whether one panel members writes the first draft of the report, asking
the other members to check, correct and add, or whether each member drafts a different section of
the report which are then assembled to the full report. A NCEQE staff member reviews the draft reportfor compliance with the formal requirements before it is sent to the university for feedback on any
possible factual errors. The final expert report is submitted to the respective Council.
The final expert report together with the minutes of the Council which include a rationale for theirdecision are published on the website as is a separate formal decision document. Since 2018 reports,
though not decisions and minutes, for institutional authorization are also published in English.
Accreditation reports are published in English where an international member had been part of theteam.
Analysis
In addition to the documents stipulating how reports are written and published, the panel also
scrutinized a number of sample reports taken randomly from the website. It may be noted that whileparts of the website are in English, the search function for reports and list of reports are not and could
only be accessed in English with Google Translate. Nevertheless, the panel was satisfied about thedetailed level of publicly available information, including all full reports and minutes of the respective
Councils.
The reports themselves clearly followed the structure provided for and thereby provide a rationale foreach judgement that is made, linking evidence, analysis and recommendations. The template also
ensured that the reports are comparable in terms of their scope as this is checked by the NCEQE staff
member. With the enhanced focus on outcome orientation rather than mere quantitative input
orientation, the reports clearly provide a step forward in supporting this development. The panel also

33/50

considered the reports to be fit for purpose, despite some concerns from the agency that fully
published reports would be used by institutions to compare and rank against each other.
Panel conclusion: fully compliant

ESG 2.7 COMPLAINTS AND APPEALSStandard:
Complaints and appeals processes should be clearly defined as part of the design of external quality
assurance processes and communicated to the institutions.
Evidence
HEIs in Georgia have the right to either appeal to NCEQE’s Appeals Council or to court. The Appeal
Council is composed of 11 members nominated by the Minister with a term of office of one year. They
discuss the cases brought before them and can either agree or disagree with the decision of theAccreditation or Authorization Council. In case of agreement, the case is closed and the HEI can appeal
to Court. If the Appeals Council upholds the appeal, it refers the procedure back to the corresponding
Council, as it cannot overturn their decision.
The Appeals Council has discussed 10 cases since its institution in 2016, out of which 2 were returnedto the relevant Council.
HEIs can comment on the site visit through a survey after the visit as well as evaluate the experts.
Analysis
The appeals procedure was found to be easily available and clear to all stakeholders. However, as in
the case of the other decision-making bodies, the panel was not convinced from the evidence that the
short term of office was beneficial to the way appeals were handled as the council members would
not be able to gain a broad overview of the decisions made. The institutions were allowed to make arepresentation at the Appeals Council meeting, and the Council members would also look at all
documentation from the process. While this process and the documentation were generally stipulated
in the Authorization and Accreditation Charter, there was some discussion as to how consistent thiswas implemented in practice. Nevertheless, the panel recognized that the primary task of the Appeals
Council was to assess whether the respective Authorization or Accreditation Council had fully analysed
all available evidence to come to their decision.
Concerning complaints, i.e. cases when an HEI would agree with the formal outcome of the process
and thus not submit an appeal, but was dissatisfied with any manner in the implementation of theprocess, this was less clear. Some discussion partners found it difficult to ascertain the difference
between appeal and complaint due to the translation of the word. Complaint does not mean adisciplinary procedure for staff as had been suggested during the site visit by the discussion partners,
nor would it be sufficient to conduct satisfaction surveys as indicated by NCEQE – which the panel
recognized the agency does in an extensive and efficient manner. The way the agency acted upon any
possible letters of complaint in this sense remained vague in terms of responsibility and process andappeared to be rather subjective. The Rule of Experts details how a procedure for terminating
membership of the expert pool can be initiated and how this is followed up. The panel subsumed thatthese were the letters of complaint mentioned by the agency. It did not, however, become clear how
this would be applied to staff members. It also did not become clear whether HEIs recognized this

34/50

opportunity of sending a letter, initiating a process as described in the Rule of Experts, or the genericcontact option on the website, as one for submitting a complaint about a procedural aspect in a
specific authorization or accreditation process. Similarly, the option of applying the General
Administrative Code of Georgia or the reference to the Code of Conduct would not seem particularly
appropriate for complaints in the sense of the ESG. The Administrative Code seems suitable rather forcomplaints against administrative acts issued by the Director. Nevertheless, the panel recognized thata few cases against both experts and staff had been implemented, so that the availability of someform of complaints processes is apparently transparent for local stakeholders.
Panel recommendations
NCEQE should make the process for complaints, i.e. dissatisfaction about the conduct of theauthorization or accreditation process or the experts or staff members involved, accessible,
understandable and transparent for all stakeholders.
Panel conclusion: substantially compliant

35/50

ESG 3.1 Activities, policy and processes for quality assurance
The progress made by the agency in the past year or so, with the redesign and implementation of the
new system as well as the high degree of stakeholder involvement in these revisions demonstrate agreat commitment to quality enhancement.
ESG 3.6 Internal quality assurance and professional conduct
The panel commended NCEQE on the use of the EFQM Excellence model as well as on the thoroughsystem of evaluations in place.
ESG 2.1 Consideration of internal quality assurance
The panel commends the close involvement of stakeholders in the revision and redesign of theagencies’ procedures.

Overall, the panel concluded that NCEQE was fully compliant with six standards, substantiallycompliant with seven standards and partially compliant with one standard.
ESG 3.1: substantially compliant
Recommendation: The involvement of all stakeholder groups in the governance should be improved,in particular since the Coordinating Council is not yet instituted and will not have a decision-making
role. A student should be imperatively included in the Coordinating Council and efforts should bemade to fill current vacancies.
ESG 3.2: fully compliant
ESG 3.3: partially compliant
Recommendations: It should be ensured that there is absolute independence from the government
in that the Minister cannot dismiss director or council member without serious reasons, thecircumstances of which should be more transparently defined beyond a mere “inappropriatemanner”.
The panel also considered that the agency should take more ownership of how councils as decisionmaking bodies are nominated under the new Rules which give this power to the Ministry.
ESG 3.4: substantially compliant
Recommendation: The agency should make use of the Twinning project to establish a sustainable
methodology for implementing systematic analysis, also beyond the lifetime of the project.
ESG 3.5: fully compliant
ESG 3.6: fully compliant

36/50

ESG 3.7: fully compliant
ESG 2.1: fully compliant
ESG 2.2: substantially compliant
Recommendations:
It is recommended to use the European Approach for Quality Assurance of Joint Programmes.
It is recommended to detail and specify how recognition of decisions or reviews by international
agencies would function, for example with regard to establishing compatibility and implementing
follow-up procedures.
NCEQE is recommended to provide more guidance to experts and the Authorization Council on how
to analyse the standard dealing with determining students’ quota.
ESG 2.3: substantially compliant
Recommendation: NCEQE has to define the details of the different follow-up procedures.
ESG 2.4: substantially compliant
Recommendation: The approach to the training of the experts should be revised, with a particularfocus on more detailed, informative briefing for international experts, also available at a longer timeahead of the onsite visit, as well as on more systematic joint training of national and international
experts.
ESG 2.5: substantially compliant
Recommendation: NCEQE should assess whether the terms of office, working methodology andvoting methodology of the Authorization and the Accreditation Councils contribute to systematicallyensuring consistency.
ESG 2.6: fully compliant
ESG 2.7: substantially compliant
Recommendation: NCEQE should make the process for complaints, i.e. dissatisfaction about the
conduct of the authorization or accreditation process or the experts or staff members involved,
accessible, understandable and transparent for all stakeholders.

In summary, in light of the documentary and oral evidence considered by it, the review panel issatisfied that, in the performance of its functions, NCEQE is in compliance with the ESG.

37/50

The panel would like to make some general and more detailed suggestions, extending beyond strictlyinterpreted ESG and/or linking several ESG, which NCEQE may wish to consider when reflecting on its
further development. These have already been signalled in the previous sections.
ESG 3.1 Activities, policies and procedures
While the panel understood the rationale behind the parallel system of authorization andaccreditation (cf. ESG 2.2), it might become sensible to reflect on the balance between effort andcontribution to quality enhancements of the two processes in the future, and when HEIs have gained
more experience in quality assurance.
The panel also encouraged the agency to consider involving international experts in the Authorization
and Accreditation Council, not least as this would contribute to an even greater acceptance of theirdecisions and trust in the processes.
ESG 3.3 Independence
The panel understood that a number of the current mechanisms of operation are governed by thenational Administrative Code due to the legal situation of the agency and its bodies. The panel
considered it beneficial, however, for the agency to explore possibilities of how to work outside of theadministrative code to enhance full independence.
ESG 3.5 Resources
It would be beneficial to gain a bit more than 10% flexibility on how the budget is distributed once
approved by the Ministry.
ESG 3.6 Internal quality assurance and professional conduct
NCEQE may wish to expand and revisit their code of conduct and the internal procedures with regard
to ensuring that guards against intolerance and discrimination are also formally in place...
ESG 2.1 Consideration of internal quality assurance
The panel would like to encourage the agency to continue exploring the opportunities for how to deal
with research elements in programme accreditation, in particular accreditation for PhD programmes.
ESG 2.3 Implementing processes
The panel encourages the agency to reconsider whether they will need such close monitoring of theinstitutions in due time in order to place more emphasis on institutional autonomy and responsibility.
ESG 2.4 Peer-review experts
The option of using online resources for expert training and collaboration among experts before thevisit should be further developed.
NCEQE should consider how often individual experts are part of procedures.
NCEQE should ensure that employer representatives are consistently members of accreditation expert
groups.

38/50

ENQA Review Site visit Agenda at the NCEQE
TimingTopicsDay 1, 30 October 2018
09:00 - 12:00 Review panel private meeting
12:00 - 13:00 Meeting with the leadership of theNCEQE
13:10 -14:00 Meeting with the team responsible onself-assessment report

14:00 -14:50
14:50 - 15:50

16:00 - 16:10
16:10 - 17:10

Lunch
Meeting with the higher education
quality assurance department
representatives
Review Panel Private meeting
Meeting with the staff coordinating the
programme and institutional
evaluations

Persons for interview
Keti Tsotniashvili as a resource person
Giorgi Vashakidze - Director
Tamar Makharashvili - Deputy Director
Keti Tsotniashvili – Coordinator of the ENQA membership application process
Lasha Margishvili – Head of the Higher Education Quality Assurance Department
Maia Gelashvili –Head of the Accreditation Division
Marina Zhvania – Head of the Strategic Development and International Relations
Department
Maia Margvelashvili – Head of the International Relations Division
Salome Benashvili – Head of the Internal Audit Department
Elene Nikolaishvili- Head of the Legal Drafting Division
Ketevan Panchulidze – Coordinator at the Qualifications Development Division
Lasha Margishvili – Head of the Higher Education Quality Assurance Department
Maia Gelashvili –Head of the Accreditation Division
Lali Giorgidze – Head of Authorization Division
Tina Dvalisvhili –Coordinator at Authorization Division
Mariam Ghambashidze -Coordinator at Authorization Division
Nino Popkhadze - Coordinator at Authorization Division
Lasha Macharashvili –Coordinator at Accreditation Division
Ani Leladze – Coordinator at Accreditation Division
Nino Gagelidze – Coordinator at Accreditation Division

39/50

17:10 - 17:20
17:20 - 18:10

Review Panel Private meeting
Meeting with the employers andpartners

As needed
Wrap-up MeetingDay 2, 31 October 2018
09:00 - 09:30 Review Panel Private Meeting
09:30 - 10:15 Meeting with the representatives ofthe Ministry
10:15 - 10:25
10:25 - 11:30

Review Panel Private Meeting
Meeting with the Rectors of thereviewed HEIs

11:30 - 11:40
11:40 - 12:40

Review Panel Private Meeting
Meeting with the Heads of QA officesof the reviewed HEIs

12:40 - 12:50
12:50 - 13:35

Review Panel Private Meeting
Meeting with the International Expert
Panel Chairs (Skype Intervention)

13:35 - 14:30

Lunch

Lika Glonti –Head of the Erasmus + Office Georgia
Nika Kochishvili - Programme Officer at EU Delegation to Georgia
Nino Kutateladze - Senior Education Specialist at World Bank Georgia, GEDDR –Education Global Practice, Europe and Central Asia
Tornike Guruli - Marketing and Sales Director at PSP Pharma
Irakli Gagua - Gagua Clinic
Nino Bogveradze – Kordzadze Law OfficeMikheil Chkhenkheli - Former Minister of Education and Science, Advisor of PrimeMinister of Georgia in the field of Education and Science
Irina Abuladze - Deputy Minister
Giorgi Sharvashidze - Rector of the Tbilisi State University
Giga Zedania- Rector of Ilia State University
Abdul Kakhidze – Vice-Rector of the Batumi State Maritime Academy
Giorgi Margvelashvili- Rector of the Shota Rustaveli Theater and Film University
Irina Shotadze – Rector of the Tbilisi Medical Academy
Davit Kereselidze-Chair of the Academic Council of the New Vision University
Irma Grdzelidze – Tbilisi State University
Anzor Beridze – Shota Rustaveli Batumi State University
Nino Chubinidze - Caucasus International University
Dimitri Gegenava – Sulkhan Saba Orbeliani Teaching University
Anano Gorgodze – Georgian Institute of Public Affairs
Nino Ghaghanidze – Tbilisi State Academy of Art
Sijbolt Noorda- Chair of Authorization Expert Panel
Andy Gibbs – Chair of the Authorization Expert Panel
Milan Pol – Chair of Accreditation/Authorization Expert Panel

40/50

14:30 - 15:20

Meeting with the Students

Giorgi Kurdiani – Tbilisi State Academy of Art (Architecture), review expert
Ketevan Kankava – Tbilisi State Medical University (Medicine), review expert
Ani Mikhelidze – Caucasus University (Business Administration), review expert,
member of the university student self-government
Tinatin Kuchukidze – David Tvildiani Medical University (Medicine), member of the
Georgian Medical Students Association, review expert
Salome Dzagnidze – Akaki Tsereteli Kutaisi State University (Humanities), review
expert15:20 - 15:30
15:30 - 16:45

16:45 - 17:00
17:00 - 17:50

Review Panel Private Meeting
Meeting with the Expert Pool Members Irine Darchia -Authorization/Accreditation expert, East European University
Elene Jibladze - Authorization/Accreditation expert, Ilia State University
Elene Cherkezia - Authorization/Accreditation expert, Tbilisi State University
Davit Sikharulidze –Accreditation Expert, Tbilisi State University
Giorgi Gvalia – Accreditation Expert, Ilia State University
Tea Gergedava – Authorization Expert, Tbilisi State University
Review Panel Private Meeting
Meeting with the Accreditation Council Irakli Burduli – Chair of the Council, TSU Dean,
Elizbar Elizbarashvili – Council Member, prof at other university, member for 3
years, previous chairman, now deputy
Dimitri Japradize – Council Member, ISU professor, since Dec 2017 member
Sopiko Lobzhanidze –Council Member, ISU professor
Levan Gordeziani –Council Member, prof at TSU
Giorgi Kvartskhava – Council Member, GTU, Dean, since Dec 2017 which was
inception of council

17:50 - 18:00
18:00 - 18:50

Review Panel Private Meeting
Meeting with the Authorization Council

Davit Aprasidze – Chair of the Council, ISU prof, 4th year
Maka Gvelesiani – Council member (student representative)
Nana Sharikadze – Council member, conservatoire, admin and prof
Rima Beriashvili – Council Member vice rector medical university
Konstantine Sirbiladze – Council , prof Kutasi state uni, 3rd year in council
Dimitri Kordzaia – Invited Council member (medical field). TSU, about 1year, on
special invitation of PM

41/50

As needed
Wrap-up MeetingDay 3, 1 November 2018
09:00 - 09:30 Review Panel Private Meeting
09:30 - 10:20 Meeting with the Appeal Council

10:20 - 10:30
10:30 - 11:15

Review Panel Private Meeting
Meeting with Strategic development
and international Relations office,Internal Audit Office, HR Division,
Financial Division

11:15 - 11:45

Meeting among panel members toagree on final issues to clarify
Meeting with the NCEQE leadership toclarify the any pending issues
Lunch
Private Meeting among panel
members to agree on the main findings
Final de-briefing meeting with staff andcouncil members of the agency toinform them regarding about
preliminary findings

11:45 - 12:30
12:30 - 13:30
13:30 - 16:30
16:30 - 17:00

Nino Okribelashvili – Chair of the Council
Nino Doborjginidze – Council member
Grigol Tatishvili –Council Member
Marina Zhvania – Head of the Strategic Development and International Relations
Department
Elene Vekua – Head of Strategic Development and Analysis Division
Salome Benashvili – Head of the Internal Audit Department
Tamuna Goshadze – Coordinator at the HR Division
Ilia Chibirovi – Head of the Financial Division

42/50

External review of the National Center for Educational Quality Enhancement (NCEQE) by theEuropean Association for Quality Assurance in Higher Education (ENQA)
Annex I: TERMS OF REFERENCE
April 2018
1. Background and Context
In 2005, Georgia joined the Bologna Process and took the responsibility to harmonize its educational
system with the European Higher Education Area (EHEA) and to implement higher education quality
assurance system. In 2006, the Ministry of Education and Science of Georgia established a quality
assurance agency - the National Center for Educational Accreditation (Order N222, the Ministry of
Education and Science), which was authorized to conduct institutional accreditation of higher
education institutions of Georgia. The system has been reformed in 2010 aiming at strengthening therole of the external quality assurance and extending the mandate of the quality assurance body. In thisregard, the Law on Educational Quality Enhancement has been introduced. By this law an independent
quality assurance body - the National Center for Educational Quality Enhancement (NCEQE) has been
established as a legal successor of the National Center for Educational Accreditation. The corefunctions of the NCEQE are: implementing external quality assurance mechanisms of all educational
institutions (higher education institutions (HEI), vocational education institutions and general
education institutions) operating in the country, on both institutional and programme level;
development of the national qualifications framework; recognition of foreign education. The NCEQE
represents Georgia in the ENIC-NARIC Networks.
To fulfil its function of implementation and development of external quality assurance mechanisms forhigher education institutions, the NCEQE continuously works on the development of the education
quality concept, creates and maintains up-to-dated quality assurance (QA) standards and procedures
and ensures relevant mechanisms for their proper implementation. In this regard, several external
reviews of the agency’s activities and the external quality assurance mechanisms have been conductedby international peers. The recommendations of the reviews have been considered for thedevelopment of the QA system.
Currently, two main external quality assurance mechanisms – authorization of educational institutions
and accreditation of educational programmes are in place.
The authorization of higher education institutions is an obligatory for all HEIs in order to be allowed
to carry out educational activities and to issue a diploma that is recognized by the state. The procedure
identifies the compatibility of educational institutions with authorization standards. The term of theauthorization is 6 years after which the HEIs are obliged to go thought the re-authorization procedurein order to continue its activates.
The accreditation of educational programmes determines the compatibility of educational
programmes with accreditation standards. Programme accreditation is mandatory only for doctoral
programmes and programmes of regulated professions (Medicine, Law, Teacher Education,

43/50

Veterinary, and Maritime). However, due to the fact that state funding goes only to accredited
programmes, 98% (1950 accredited programmes) of all academic programs are accredited.
To ensure the continuous development of education quality and the consideration of the external
evaluation results by the HEIs, the NCEQE carries out follow-up monitoring procedures for botheducational institutions and educational programmes.
Furthermore, the NCEQE supports the development of internal quality assurance mechanisms of HEIs
through providing various capacity building activities, including provision of guidelines, consultations
and workshops for HEIs.
In 2015-2017, NCEQE revised the system of higher education quality assurance to strengthen thedevelopment-oriented and outcome-based function of quality assurance and to ensure its compliancewith the requirements of Standards and Guidelines for Quality Assurance in the European Higher
Education Area (ESG 2015). Specifically, the quality assurance standards and procedures have beenrevised; formal procedure of complaints and appeals has been introduced; students, employer
representatives and international experts have been involved in the review panels; evaluation reports
became publicly available.
In order to foster the proper implementation of the revised QA system special programme the NCEQE
carried out pilot evaluations of HEIs, provided capacity building activities to HEIs and experts, anddeveloped review process guidelines. Special programme under the project “Study in Georgia” hasbeen designed to support the above mentioned activities.
From 2018, the NCEQE has started the system-wide implementation of the revised quality assurance
standards and procedures at Georgian HEIs.
The NCEQE has been an ENQA affiliate since October 2013 and now is applying for ENQA membership.
The NCEQE is also applying for registration on EQAR.
2. Purpose and Scope of the Evaluation
This review, will evaluate the way in which and to what extent NCEQE fulfils the Standards andGuidelines for Quality Assurance in the European Higher Education Area (ESG). Consequently, thereview will provide information to the ENQA Board to aid its consideration of whether membership ofNCEQE should be reconfirmed and to EQAR to support NCEQE application to the register.
The review panel is not expected, however, to make any judgements as regards granting membership.
2.1 Activities of NCEQE within the scope of the ESG
In order for NCEQE to apply for ENQA membership and for registration in EQAR, this review will analyse
all NCEQE activities that are within the scope of the ESG, i.e. reviews, audits, evaluations or
accreditation of higher education institutions or programmes that relate to teaching and learning (andtheir relevant links to research and innovation). This is regardless of whether these activities are carriedout within or outside the EHEA, and whether they are obligatory or voluntary.
The following activities of NCEQE have to be addressed in the external review:

The authorization (and re-authorisation) of higher education institutions. It is an obligatory
procedure in order to allow higher education institution to carry out educational activities and toissue a diploma that is recognized by the state. The procedure identifies the compatibility ofeducational institutions with authorization standards. The HEIs shall apply for re-authorization in
every 6 years.
44/50



Accreditation of educational programmes. The procedure to determine the compliance of
educational programs with accreditation standards, which facilitate the development of quality
assurance mechanisms for education quality improvement. The term of initial accreditation is 4
years, while the term of the reaccreditation of educational programmes is 7 years.
Furthermore, the self-assessment report (and external review report) should also address thearrangements for the recognition of the external quality assurance activities carried out by otherquality assurance agencies and how NCEQE ensures ESG compliance in cases the agency is not
EQAR-registered.

3. The Review Process
The process is designed in the light of the Guidelines for ENQA Agency Reviews and in line with therequirements of the EQAR Procedures for Applications.
The evaluation procedure consists of the following steps:









Formulation of the Terms of Reference and protocol for the review;
Nomination and appointment of the review panel;
Self-assessment by NCEQE including the preparation of a self-assessment report;
A site visit by the review panel to NCEQE;
Preparation and completion of the final evaluation report by the review panel;
Scrutiny of the final evaluation report by the ENQA Review Committee;
Analysis of the scrutiny by the ENQA Board and their decision regarding ENQA membership;
Follow-up of the panel’s and/or ENQA Board’s recommendations by the agency, including a
voluntary follow-up visit.

3.1 Nomination and appointment of the review team members
The review panel consists of four members: one or two quality assurance experts, an academic
employed by a higher education institution, student member, and eventually a labour market
representative (if requested). One of the members will serve as the chair of the review panel, andanother member as a review secretary. For ENQA Agency Reviews at least one of the reviewers is an
ENQA nominee (most often the QA professional[s]). At least one of the reviewers is appointed fromthe nominees of either the European University Association (EUA) or the European Association of
Institutions in Higher Education (EURASHE), and the student member is always selected from among
the ESU-nominated reviewers. If requested, the labour market representative may come from theBusiness Europe nominees or from ENQA. An additional panel member may be included in the panelat the request of the agency under review. In this case an additional fee to cover the reviewer’s fee
and travel expenses is applied.
In addition to the four members, the panel will be supported by the ENQA Secretariat review
coordinator who will monitor the integrity of the process and ensure that ENQA expectations are met
throughout the process. The ENQA staff member will not be the Secretary of the review and will notparticipate in the discussions during the site visit interviews.
Current members of the ENQA Board are not eligible to serve as reviewers.
ENQA will provide NCEQE with the list of suggested experts with their respective curriculum vitae toestablish that there are no known conflicts of interest. The experts will have to sign a non-conflict ofinterest statement as regards NCEQE review.
3.2 Self-assessment by NCEQE, including the preparation of a self-assessment report

45/50

NCEQE is responsible for the execution and organisation of its own self-assessment process and shalltake into account the following guidance:Self-assessment is organised as a project with a clearly defined schedule and includes all
relevant internal and external stakeholders;

The self-assessment report is broken down by the topics of the evaluation and is expected tocontain, among others: a brief description of the national HE and QA system; backgrounddescription of the current situation of the Agency; an analysis and appraisal of the current
situation; proposals for improvement and measures already planned; a SWOT analysis; eachcriterion (ESG part II and III) addressed individually. All agency’s QA activities (whether within
their national jurisdiction or outside of it, and whether obligatory or voluntary) will be
described and their compliance with the ESG analysed.

The report is well-structured, concise and comprehensively prepared. It clearly demonstrates
the extent to which NCEQE fulfils its tasks of external quality assurance and meets the ESG andthus the requirements of ENQA membership.

The self-assessment report is submitted to the ENQA Secretariat who has 4 weeks to prescrutinise it before forwarding the report to the panel of experts. The purpose of the prescrutiny is to ensure that the self-assessment report is satisfactory for the consideration of thepanel. The Secretariat will not judge the content of information itself but whether thenecessary information, as stated in the Guidelines for ENQA Agency Reviews, is present. For
the second and subsequent reviews, the agency is expected to enlist the recommendations
provided in the previous review and to outline actions taken to meet these recommendations.
In case the self-assessment report does not contain the necessary information and fails torespect the requested form and content, the ENQA Secretariat reserves the right to reject thereport and ask for a revised version within 4 weeks. In such cases, an additional fee of 1000 €will be charged to the agency.

The report is submitted to the review panel a minimum of six weeks prior to the site visit.

3.3 A Site Visit by the Review Panel
NCEQE will draw up a draft proposal of the schedule for the site visit to be submitted to the review
panel at least two months before the planned dates of the visit. The schedule includes an indicative
timetable of the meetings and other exercises to be undertaken by the review panel during the site
visit, the duration of which is 2,5 days. The approved schedule shall be given to NCEQE at least one
month before the site visit, in order to properly organise the requested interviews.
The review panel will be assisted by NCEQE in arriving in Tbilisi, Georgia.
The site visit will close with a final de-briefing meeting outlining the panel’s overall impressions but
not its judgement on compliance or granting of ENQA membership.
3.4 Preparation and completion of the final evaluation report
On the basis of the review panel’s findings, the review secretary will draft the report in consultation
with the review panel. The report will take into account the purpose and scope of the evaluation as
defined under articles 2 and 2.1. It will also provide a clear rationale for its findings with regards toeach ESG. A draft will be first submitted to the ENQA review coordinator who will check the report forconsistency, clarity and language and it will be then submitted to NCEQE within 11 weeks of the site
visit for comment on factual accuracy. If NCEQE chooses to provide a statement in reference to thedraft report it will be submitted to the chair of the review panel within two weeks after the receipt ofthe draft report. Thereafter the review panel will take into account the statement by NCEQE, finalisethe document and submit it to ENQA.
46/50

The report is to be finalised within three months of the site visit and will not exceed 40 pages in length.
When preparing the report, the review panel should also bear in mind the EQAR Policy on the Use andInterpretation of the ESG, so as to ensure that the report will contain sufficient information for theRegister Committee for application to EQAR.
NCEQE is also requested to provide a letter addressed to the ENQA Board outlining its motivation
applying for membership and the ways in which NCEQE expects to contribute to the work andobjectives of ENQA during its membership. This letter will be discussed along with the final evaluation
report.
4. Follow-up Process and Publication of the Report
NCEQE will consider the expert panel’s report and will publish it on its website once the ENQA Board
has made its decision. The report will also be published on the ENQA website, regardless of the review
outcome and decision by the ENQA Board. NCEQE commits to preparing a follow-up plan in which it
addresses the recommendations of the review panel and to submitting a follow-up report to the ENQA
Board. The follow-up report will be published on the ENQA website, in addition to the full review report
and the Board’s decision.
The follow-up report will be complemented by a small-scale visit to the agency performed by two
members of the original panel (whenever possible). This visit will be used to discuss issues, based on
the ESG, considered as of particular importance or challenge by NCEQE. Its purpose is entirely
developmental and has no impact on the judgement of membership and/or compliance of the agency
with the ESG. Should the agency not wish to take advantage of this opportunity, it may opt out by
informing the ENQA Review Coordinator about this.
5. Use of the report
ENQA shall retain ownership of the report. The intellectual property of all works created by the expert
panel in connection with the review contract, including specifically any written reports, shall be vested
in ENQA.
The review report is used by the Board of ENQA for the purpose of reaching a conclusion on whether
NCEQE has met the ESG and can be thus admitted/reconfirmed as a member of ENQA. The report will
also be used for registration on EQAR, and is designed so as to serve these two purposes. However,
the review report is to be considered final only after being approved by the ENQA Board. Oncesubmitted to NCEQE and ENQA and until it is approved by the Board the report may not be used orrelied upon by NCEQE , the panel and any third party and may not be disclosed without the priorwritten consent of ENQA. NCEQE may use the report at its discretion only after the Board has approvedof the report. The approval of the report is independent of the decision on membership.
The Chair of the panel shall remain available to respond to questions of clarification or further
information from the EQAR Register Committee provided that the ENQA Secretariat is copied in all
such requests.

47/50

6. Budget
NCEQE shall pay the following review related fees:
Fee of the Chair
Fee of the Secretary
Fee of the 2 other panel members
Fee of 2 panel members for follow-up visit
Administrative overhead for ENQA Secretariat
Experts Training fundApproximate travel and subsistence expenses
Travel and subsistence expenses follow-up visit

4,500 EUR
4,500 EUR
4,000 EUR (2,000 EUR each)
1,000 EUR (500 EUR each)
7,000 EUR
1,400 EUR
6,000 EUR
1,600 EUR

This gives a total indicative cost of 30,000.00 EUR VAT excl. for a review team of 4 members. In thecase that the allowance for travel and subsistence expenses is exceeded, NCEQE will cover any
additional costs after the completion of the review. However, the ENQA Secretariat will endeavour tokeep the travel and subsistence expenses in the limits of the planned budget, and will refund thedifference to NCEQE if the travel and subsistence expenses go under budget.
The fee of the follow-up visit is included in the overall cost of the review and will not be reimbursed incase the agency does not wish to benefit from it.
In the event of a second site visit required by the Board and aiming at completing the assessment ofcompliance, and should the agency accept a second visit, an additional fee of 500 EUR per expert, as
well as travel and subsistence costs are recoverable from the agency.
7. Indicative Schedule of the Review
Agreement on terms of reference
Appointment of review panel members
Self-assessment completed
Pre-screening of SAR by ENQA coordinator
Preparation of site visit schedule and indicative timetable
Briefing of review panel members
Review panel site visit
Draft of evaluation report and submitting it to ENQA coordinatorfor pre-screening
Draft of evaluation report to NCEQE
Statement of NCEQE to review panel if necessary
Submission of final report to ENQA
Consideration of the report by ENQA Board and response of
NCEQE
Publication of the report

48/50

April 2018
July 2018
Early August 2018
August 2018
September/October 2018
October 2018
November 2018
By Mid-January 2019
February 2019
February 2019
Early March 2019
April 2019 (depending on the date of
the ENQA Board meeting)
April/May 2019

ENQA

European Association for Quality Assurance in Higher Education

ESG

Standards and Guidelines for Quality Assurance in the European Higher Education Area,
2015

HE

higher education

HEI

higher education institution

NCEQE

National Center for Educational Quality Enhancement

QA

quality assuranceSAR

self-assessment report

49/50

DOCUMENTS PROVIDED BY NCEQE

SAR NCEQE for ENQA Review

Annex1. Law on Educational Quality Enhancement

Annex 2. Charter of the National Center for Educational Quality Enhancement

Annex 3. Charter of Authorization of Educational Institutions (with Authorization Standards)

Annex 4. Charter of Accreditation of Educational Programmes (with Accreditation Standards)

Annex 5. Rule of Authorization and Accreditation Experts’ Selection and Activities andTermination of Membership of Expert PoolAnnex 6. Quality Assurance Mechanism of the NCEQE’s activitiesAnnex 7. Self-Evaluation Report Template for Accreditation

Annex 8. Self-Evaluation Report Template for Authorization

Annex 9. Experts’ Evaluation report Template for Accreditation

Annex 10. Expert’s Evaluation report template for Authorization

OTHER SOURCES USED BY THE REVIEW PANELNCEQE website: www.eqe.ge/eng

The following documents downloaded from the website:

Code of ethics of experts



Authorization of Higher Education Institutions



Guidebook



NCEQE Strategy 2016-2020



NCEQE Strategy Implementation Action Plan 2018-2019Sample authorization and accreditation reports

The following documents provided during the onsite visit:

Excerpt, Article 17 from the internal regulations (translated, in Georgian on website)Overview of number of experts in expert pool

Income and expenditure on higher education quality assurance activities



List of proposed candidates for the Coordinating Council



Excerpt from Twinning Fiche for project Strengthening capacities for quality assurance andgovernance of qualifications (confidential)

50/50

THIS REPORT presents findings of the ENQA Agency Review of the National Center for Educational Quality
Enhancement (NCEQE), undertaken in 2018.

2018 ENQA AGENCY REVIEW