Repository: JohnSnowLabs/langtest (language: python)

Issue:
Add new Factuality test
Follow this approach: [Llama 2 vs GPT](https://www.anyscale.com/blog/llama-2-is-about-as-factually-accurate-as-gpt-4-for-summaries-and-is-30x-cheaper). 

The idea is to give the LLM model two texts, one which is factual and accurate and one that is not, and the model should say which one is the factual one. 

Check the link to see if the model data is available (prompt and texts), otherwise create an issue to build this kind of data in future.


Discussion / hints:
for your convenance : 

Factuality test Data: [Dataset](https://github.com/anyscale/factuality-eval/blob/main/resources/datasets/val_sentence_pairs.json)
Factuality test prompts used for experiment: [Prompts used](https://github.com/anyscale/factuality-eval/tree/main/resources/environments/answer_first/prompts)
Evaluation Notebook : [Notebook](https://github.com/anyscale/factuality-eval/blob/main/factuality_evaluation.ipynb)

Fix patch:
diff --git a/demo/tutorials/llm_notebooks/Factuality_Test.ipynb b/demo/tutorials/llm_notebooks/Factuality_Test.ipynb
new file mode 100644
index 000000000..ff8345d1c
--- /dev/null
+++ b/demo/tutorials/llm_notebooks/Factuality_Test.ipynb
@@ -0,0 +1,1400 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUgAAABcCAYAAAAMJCwKAAAgAElEQVR4nOy9f5gcZ3Xn+znnra5pjcfKZCyNfqDIQgghZMdxZMfGxpbbwhjM2g4h2Ak/Nol3Aw5xEsLu5eHh8vCofNl9uFluLhiwhUi4zib3ZomcZBMgARsjt4RxbGIritcSsiyE0GpleSQLMYxHPd1V59w/qnq6Z6ZnNJJG/Ej6+zw9PW911fueeqvq1Pn9CucASZJokkzZaudirC666KKLcwWZ+y4TveyWJeW4/lKZYYD5mI2m8+YdH61Wk3Tux+uiiy66ODeYYwaZaKUysNSI7xSVtfj4MCPi9t8WLhzY+sADt9fndswuuuiii3ODaO66ShQSM7lvvYj8B6A8/pMIiM4/evToTuDI3I3ZRRdddHHuMIcMMocgC9ysFwx3DBzVyFzCQBpF8VyP10UXXXRxrjDnDBJygdFyl4wiTS3egJPnYrguuuiii3MCPRedem57NHBk3A6pwLxzMVwXXXTRxTnBnEmQSZJ/xP2gaDjhrv00vTSigB12tVqSJNrcf/p+uiFBXXTRxY8ec+7Fvuqq+f1RT/ktgl40PogwbKn/XQgv7KhUsJwBJjNIr10G2UUXXfzocU7iICsV9AfnL4k5nG85//zYKpXv1pMksStv+uT8eKy0RtyWqU9U8U1cU5e9Mb17qtU7anNPWxdddNHF7HEOGOTUTJpKBa1UsC271kYLjh79zyL6bnefP3F4b5JzxLEPvrhw4Z/v7sZMdtFFFz9CnBMGORW5On1V5YLVsUT/CNJrlnXcUzXg+JfU7c5K5ehQ1x7ZRRdd/KhwTsJ8JqMpTW7dzlJc+swykBZ3HpcdAfcMkVAGLVerKHl8UBdddNHFDx3nJMxn2sHMFYrEmrbtPyQxtosuuujitPBDlSDXbwgqDo4grUTtCRJkF1100cWPC+aIQc4uZMdMLAhtzDH/lo7KdhdddNHFjxZzwCATXbuWCNZO8/sWBgdfUvhuCh75hN8mM8P2djfKp4suuvjR4iwYZKLXvq7/YrGeD7jbIBxF3NskyZZ/JTc9LkyBBdP5XNxBwETV8OwwcKJSwarVM6ewiy666OJscEb6bJIkWq0uXOkS/ptqaZ1ZSqsoxQxwU/f28J7Jxzil6LwnG/aDD2zf+rtbz4S2Lrrooou5whlLkCa+LmjP8ix9KXUkEloWxBm+TaTwnDsmok+L6iHcIxcxaBzP0h98bnvlxe1szetLnu0JdtFFF12cKc6YQbprjLgiolKECzXlwVN9Fz2kmdumyPyhNLhGmRhEI9XqnceongFzLIpg0A0s76KLLuYILQaZJAobIZFZMphsgnQ4W7g7ICaAqp2oXHfs4K5dREePthsnZ2BySdPOWS2+K5bTvLG5rcsgu+iiizlBziCTRyIWDpY5ursO5PnPic8QunM3ofgvZ46T2eSp2tB04iRJYkmSpDOmFCau44x77e6II3GZ0s+U0bEyvq+PTc/2Ic8tw5fGJL5l9ky+iy666GJ65AxyydJVuN7OYh/lM88OIQwjz42QygjKMJ6OYlajhzqhd5Q7qFPJO/Ai7Lv5fx7VOHO7CfdZZPJsPtwLe9fxmb2D4H286IuJWYTqAvS8BbgsRmwAGCTL9gFb5mhuuuiii3/lyBlkqsuZN+8OsvogIaqhOgqhRikbJUtHca2TpaM0pE5afzBJNn5m/bb7VGkP8p74/3TtcSapBhODIjvDvj9I+fy7kbCGtF7GrBfPYtwUc8vXd3AIEdC5AEYXXXTRxZkgZ5Alt9yg6BH1sX5gfsHbNOdnriBQ7jVOvpRWqH72rHVYY3bGSytFNBqLkXSQrFFInN70hBffbmiYZYdddNFFF7NDIUECJcgZjytNxtiEA7iRpYqQTu2mubPMsi2AIGKz5LMCmOKmHeMtu3yxiy66OAeI2v6eIthbirVlRGGyq3imlMHJ7bbM60ICzMuatSrsTlmXRrFZqeNddNFFF3OIXEXtIBNOz5CauvfZQ0TqANXqRH47qyK5XYbZRRddnGNMlCDbMUWY7MyR2r3Ys4XjiKC4r61UPnMQsrJpi0lm+olDpfTE4Wo16cS6p6Gviy666GJuMZE1+mTD4/RcyFWsGcRzOpCWAKogHzGyjwATdPbg8QF06d2Vyv2fn75WRbc0WhdddHFuMclJAy3GM7lG4xSHSwp5QLa7W3uwT4t1easHkem1cqHVrWMi0XIXeY9Qa/LHtmOno+cnH801wydt6wa9d9HFjwgdVOxTOVya8N2W1YdE4wXi2YxH5BFERidm5u75/sVPDmAZIEsta/QC9YnHdex9GhrPHJ2YVbH9HDCsRG+6aaCvWg29k3+pVDanlcrzx//lMMr2eW2d08SVMP+lnOuPEdoz485Vptnk7LvTHSdxhbvJ04anw91nXm+hSV87XaeYl4kqdrsXe4oGOy7iWZWKVbJtu2HwfZlnG8VZPC1RCuLgbgMg/ePVfMaHLAZpfakI5gBxTOvHSUzwHGrY0zHHczXWU08tKZ8YyX4f918uwt5VwAwipfF0tbrkvUmS/EQzyZwBJkYClSo6NFRELly0FtjNll1Q1P+05vz/JJ9vF2eARGxqrYV2VIqaC8nE9ONT9lvUmWj2u2VXG9/bDbuHLO+bKf1Ob4OcUqpxIiOrVLAk+e2HIdl62WVLykuXTkfd8wCcGB78UAjRfzCrRyAzVBGapTR4jpjjbbdtiavVY+sybIUIRhaADIJHiB4DHprrMYeGxqK4HF6uIbrYLVMpXgiRBixr1EulenzKTn5skWilglarS/qvrty7LFTlNSby6gWLfJkg/Rw7rrB4FOG4kR1av97/6aGq7CXWw5VKcnxGR10Xs8Omb61A9l0OGXhQPv2tnfzOq/fOWf/JIxFLll2CPbsq3yCK6yj3f2c7d7z8xCmP37Ir5lhpGZEuxp5dCroAedl8JJQR78ElxTmJ7x0G389nnjuI7B0i8eP5+DMwysSVnzown/i5FaitI7rwSk74UpA+xFPcj7P0woPw3C42P/c0YfcBEj/R7HN6RuU+KS6yybgKKRVyzpwk9tRTjD711LQUKsC111nqba6Yyd7vZnvWPvEp9J09KpUkOjR8qC/WeXeKh7fnGToOLghR5GZPcg4Y5Lx5wTL31C2z3BSRM0jLR09H53rAHwKaUmC1urA3w25Q4ZYS4Ro3WyUiKqJ4YcMW0DyyIeBqtZLqARq+AwY/BTz+Iz2Rn2Q0JSd/7mpCuAejTKlkYB8C5oZBJolywZJBotIHSeVW8BSIEB2hkd4BfKHJJzof78rRby9nXvmjZI31CPNxi0GLpBAthCEDF0PCMCE6hNsOFu39Mg39exIfmZZJLn52HRq/DS29kbSxGhFFFEQUHBzDHUxSotJBTP+SZbs/1mSSE+MgRVpSZJP5TG5PqEp2ahWoZVcquivY38QCFq32KVleJ/rm0ATZM3aeQkCQCCd2J3aIEVVkJsn37CCtOyEPgZrgiPrJxBe/uKScuX44aM/HwX8NfBU47hlmDSyr5x+r45ZinoEQ46zGeKuJLYcfrsnjXxaaaqUoqhEiMVEMOoPD9ExQ0lVIuJjcfFYGIkLUj+hNwKn5hKS9qCwDGaD5rIWIfBGWDDzL81OiHiWEftzW4PZOeno/TmQbedm+pR2rj21+9hqi8iZEfhv31WgUIZr32RiDtFgJQRVEIpxVGOsIvdOo2DBVahxvnzkXShL42rai+0nGw9MNE+pM31w7aQzM8WbON27F2+aHgJ9873zTrnre+endIfT8dpaNxTiKoHnWapvtuWi3NRRxQ+WAethd9Ne1RZ4NJrAOn7uKqYkra3dHHLN1pPXlxeJTxRgZmN/A//vcfN75yuHpO7kb5J2FFJfm6cRwgKzxNwj/E6eGiaLWh6SvxFmPllbgBo2xBcQ9v0Wj3s/CAx8i8aFxO+aSfZcS9XycrL4OMyOUFLLDGF/CfRduI0BMlr4c90twW8d5fQsYPvY1vvuq4dxZNNmL3ZTOxnmYTGqfBQwIs+lqMmMYyw+cvEs7fXMNV/WiMlBLqJbTZ+b/SrFlF9HCkfR3Qii/O01PxiIStU+d5Kq1tiWdGoKKY/nLCEXYWS8xVKkkUdcOORdwxl/ycyk/vhAW0Ft+HZmVUVXS9CuUoktxHyREqxitryfxvwdmthU26z3kmtROTD7KC684NuWY+7/TT73+a2j0XsxXkDViSvHtZNn/4MIDnyHxlEXfHsDlA5hdipmhoY5nW8jC3bzn5QemjJ24sujAcn7w4luw7AtTnTQT4iCZJtJnbpjDqXtpqdo5q+yZ0OrYyU+usNUBk+M8f7JQLOi2lhDdlqVjfcJEdU5EUxE9CLbHPT3miKlIHxIGUF2M23KgTJb+c2znDXdXtpwrTHSyzgkSMe57bjlZdmmxxRC/n6h0F5ktQAOkfhNUv0Jy/Wm85DwizSKuQ0naH+674bsrhlny/B+TvZQSlT5CI+1HrZcQ3sBIbQtUh5CfWUccX06jDhqBsJVG9hGGXnFw2kLgL6w4SCL/9+TNp1Gs4sxQVAxXhe+rBMuQIrB8qoMGwAUTFBEZcer5pJ6qNNo5oHvSALPeczycZdK24vuslZvJ/Z+q79kEn7diECfHJZ4+vdUqmrpfEcxX57p06zeRAOJfERu7B0r76uXGcM+YGMRlPOuzLBuUwKVo6UqX8Pj1679bb94/pzqHs6F5ch/5N0yOx5yu/5lspDPRM/m4TmOeaozZn2+bdjgXKnYzHCYK1yC6ODdLZUOkPEpmr8eya8hSRaPXMPiy5SR+4LTjIrdhU45JNirPL6mx8MBfo+k7CKXX5GdkawjxAi5ccZyxxsWk9aW4QVwe4eTI3zH0qoP58dPQMA3j7BzmM9lDfJYe4yRJ7NprP/Gwp/V3hKh86cyKtqu51zJPv9DosSPAYO5JnkRnRw/73KEps+aUztx/O5NKinbTNzXl+5QPcbOo8ERUq2iSJIz3P8n5Nf3DO3176kOXKLPstxOSJNEvPzHQW66Fi9ysb9zmSG6gcLNhj/QDgeN7Ad5wVf6oVquMAMe2b0/23XbbliePHv3eFqE80hw3/y5oSzoO3U7EeJhFqyrU7BaBa55ra15a85Mk01/D6embpRNz/LgZmanl3uDmhsljnQpzrJWMMxq/CRUgMpxvsqh+jO/V/wcS1fAsJu5dRnbychLZf0rypqDDGlOJ5PNwdOMQS57bQ6nnNaR1cPqwrJ8fSMw8/Rncy+ApwgjoPujAbDuez0RMVLHbvdhNJjQeG3l2TOjrX//9pyuVe/+NWe0t7lZkjDTvvxZt4sFcbU9w2f7El39vhJvfNJinNLbR1ZG+uUXrwW6Xb6dWLE+SRLfsWhsNHj0yuH7Dp1bLtvCaRwivuA4WQBY/4jricOhasn/m2vt2fPnL6QFg+HSlnaEh9KuP9i+9Juu5YSty5XUbfCnmPLJN9nuWfSPL0scrleRwXhkp77dS2bQiwy/11FJVVVOxrdsye+3rP7Xz9a998UheZm7higy9/LrruQp0BdssAj3yCPbPlcq926vV3j1JktRnS2vISmURHURzb7XguIuJBpzs4Ne/dmRPMXPtqvN43xddtDtNkuRYs33ZZZt7zz+/foUZ860qputVATz69KEXLxh8ZvDobhsbmz9fe3rWbt2u16x3+XnB5rNBRrZW/cA1lU8+GNGzE5ITM9kyK5UkeuihRQPr19+76pFtevl118urcJaSe2VrW6scuZb0Wat86tFqNT5QqeT9VSr3l2H0cjMbaNJnKqbmCvcc2779vY91GqvOwou3bpPl11TMqIKuV0313oOPVe/aOXX/+8uZ1i6Rbb6Y9cWEVc2iikZZ+OTer3/t93af+so0X/fMnQ3yvj2X4H4NaUMRMdz/jtsvqrP52R2E6ABuq0nTAcRfxyef+wrHV00fjnMmj7Fbffx/kTpRGOWkKm5Riy+IgkzJUJstpqYaTpYUJ4f7nAWq1buOAPedar9WDF2HHzvSdy6NkNImQU50FiVJol/9av+yhfHRm116flHcLgcGkOZNEEAEcVdcUonCgbLKX1+74dN/Ua0e250kSZ0OaB9RALFQvmBwwVvUone523rRkN/iWkjiwm9GpWg7LL4HfusrkEuYW7dlG5Tojzx4DUHVzUTiUW003l+tLvxLM26UEL1PsHUQehGseY754pPRPhi9p1rt2wIc60DqjBhfkUhcPU9HXXbttYMXv+51Q8/kNHZUVydsmzcvW+we/YEIl6q4oYCLikd/0//9F38XLlhe6gn/HuRmcVla1CzNRxZXNfl3HvE3kl2wqVJJdnZikle94Y8HsrGxDaUe/SWMG9xYIKoTGEkeiqcaiR5w2Oos+KvLLttchXqvubwHid6q5PSpuEnQ2C3aWakkV7WPmSSJfvUbFwyW0ujDbtnNiqSIqASNStjDwE3ttFUqj0Rp2LU8ePRRd7+6SZO6mmsoq/EeYBYMsg1z5cVWuYFSOSIdM5BDYE8CUPf9SGMvImuwFOLyJdjoCrj7mbkZeCMs291PI1pNVoTqiB7ETx6j96U6dv4xJKQgkGXzwS7jwgMPkST1001TnL4e5GScczvfRJyWLekcO2m8k/yfJFqtXrA6RPGnIPrP4De4eb+54Vkzxq+BZ3XcU8AjsJUov68S3Zux4M1ffGpJOZfiOp9MMeWxpPZOJXwUZL27q2f1vN+sgWcNwMuOvxENH69U7nvNuBqdaU01KEgZJ0aIVUOs7ksz+A2Nev4Q/Grce90LWpv9muFuKyF8xCj/1k03fXL+bOIR43qtbm7H3a3wSkPLbCD9ov7Rr1YHr9iya+2kJYc7I4rE0JCiGmHEOLEEjZQwX+q22qV0r4j+O5ylbpm25iWPrQTvF5O3u0QfzbKB1ZP7r1TuXRzX7UMq0cfBf9VhgWOYNcav43if7ubmy8F/TSW+5/zz7feGFv70sKg+JSKG5/RhRSygyKpG44LBibdNYpr5MlFdKSqtawORO5dWKpsXTKRvm6mzGMIyEYnHx4AyeE1cpkioM6KIvT4rJIly/3f6gdcXy6AoIjtI64dJXHnx+SHcniCKR4EU95WIrJ05x7oN0wljSaLjtsK0VKHUs5YsNZAU9ypmx3j+sjruu4ii44hAWu8lKr2Z2tjVrL0tym2ns4+rzXecHObzI8aPX9zb1HmpVC9YnRE2icrNbul890wR0yYrLbJFtJ25upu6W+yZXy4e/vC8kcbNUyWacS++uhuOrBb0P7r7cstSLVxammcESB5bKK7uZu7Zmgzf+NBDixbkc+i1PI7eQUxx1KwRu8htKuH95o1lZinuZjjmbX2Cq3umjs8XLb3rByd1PcwmaPv7I0L2zyI6MjHeFXAzRG6MNHzugqGhjZXKp9aQd2rkJocpfTcaYybjBUscxNUtU7N0tbr/IcgVbhYVvNha8yKKgONq1oiRaL2WSu+f2HuirtHHReTd7tni/HwzBVcBXFAR1bbzUMSa46+QEH9w4dDQ73iWPSOqRxAMseJ6ZIjo/FJJV7aGK87RwnJ3W+qeX5e2/QfNGmsLm2lrPlJdhtsCt2J/DNEA5nvghT0zX49JmCsnTb1+MaXyGiw1oEaWfoOFHM+LSVyfYjwOHMctIksHiEpXMbCvb+blpAtMJ4s1+cLi564h6vkAWTqAqqL6NHbyAY4+MAoYFu3A/BmcCDMQ1hJKH+NY/MbChpnHSs6Clok7zCgl/ngwz444x8JtK+snI0kSrVQ2rXDCx1R0vecXILeL5a/nVELphIjsNfc9IcRDImEiE/RMRWWxEG2+9nX3XXLyZKaTw2HGz0noBe/L/1VUo1SQnKG17SqCmmdpFHpeE+L0LUmSqKnXJ3QoqHtWBrnULFuGmZL3aaKKeMs+JCKIiLplkWe2LEjpjmp14eBkp087kiSxSgUT9+2CPi46yd6UF0lWz7I1IcT/u0v0j9dtuO/Prq3c9+bXfnXJsi1b1kaTmWSppOZNHWe80ImD+EoRvcIsNQRVVUSDFT/bhIQrcfWsHrn7r61ff+/VkOhll23uXV8Z/AOV8KtZNtYLFo2fN2IaolGVsB9nt4TosGioC0W/goJFWVbrDaXeD6Csc2cvIupe3C3uphppBs0QGBLy1Etcf8GzbAGeL4ZXVLMy1aAeqOQ25MSqVbRaXdiL+s+6Zf15VpxAca+4yN9Xq0n6Q800ShKF65RM14MMgqRE8X5UHmf32nSciVn9ScZGnyaKQQKIVuixaSs2FCgW4ZMyJZayaPEyNn1rBfftXcnmZ9fw2b03sOQ7mwjRf8fSy9EIgj6O1d/LnWt35IxPjLtW7SPLPkb5vL2okku5cimBv+Wz+/8rn917Awt3D0JVT8UoO8dBdsT0XChx1yLwfE6QnKtyTKeBiT5yz62CrrlDRl+8WQjXFA/nuKoooiaqO71R36QavknGaCb1derhXaJhvVsWk8cwqVlmqqV+Se0DIZTeZ3gqjk728I8nZmrY75buMOe4qi4vJKeBPPOkuZdHZo35SrjuoccW/XUkmRVse1IuRe52EpW6oI+aNQ4gUtYQXeKWXTJZzc+7tyvAlkFy5NRe4Rf3Zb7gc0HjNe4sds90vB6ooI5hWcMQ6ROJ3i6kb45i/+bCRcf/qlod+AJwqOmpbzTESrGk3kZ38yxwN5HIVGSve7bTzU5I0NWIrMOy/lawQ26nVonVqN8CyWPnnffpimjp7WluP8sZjjuCGnAo8+xz5tnfSxSOq9sKcf6tiLzV3fpaHmGP0sbYAkF/CU+HNET1jCxu7w+4qDlfCfDahs0v9ZTWuhvuaZt06nlMs8vP33LL5t4vfvH5WrWKXX2j9pbSsAo3xX2cRvdsGPWvz3wXT4OzYqcb4WX7FuPhKtJ6nKuxjd00xiZ6qe+6aIRNzz6I6M1kYyC6CgmXksie6SvxCGCgcjla2gyhmTgQgffhtpigfWQpwGG88RUyPs6RVROl6MSVIzzEon0fpjzvD2iMrSgkXSPSd5Lpmyj1PsqSpV9G9lQ5fGR/EfIwTbmzM1GxN26EJOETu04ul2dH3+S/IhHuhoQzn37PDAKf+NWxR39/Tc/TZ9zPHKAV4tPGpAQbPHpk0CX+JfD5tN9qriYiJ9wb/3HDhmOPNjfv2rX20JEXXzyo5veAXOHuxUPratYwDfE1sTQuMbfc09tWetidIutEdpqnH80auj2ObbQRxgaiLHqnavR+t6y/RbXg5mgUrQhZulhdzCfFIgKIYwh1N/usRX5P5DIE9ahhsiYS+SOQi/OiGQV7dVPQxYJeDDyZJFPDh5oowmSoVuVLnjUGRMNHRaI+LyQ9mhlJuRqf21CFPjeviMrlaPn69Rs+/alq9dhjlQo0GuDixaJtE9ITTTQC829CfaNQ3yk6r4bbYkPuFA3vxrK+1jUS3DMQW1epbF7gkv0i7oMTcyDERMOwe/qpejn77BNfPj5S/HCgUhnYax56VUu3uzVyVb4ZDKa6yiwbVbeaIHFz3twzcF9dqfzU/GolGSZJrFTZNGDua5quxXH2KCi5mr36e99rLAP2QWKa3dcHvpKiDB5Cs97CHjLfe0axn2cjfiRibPrWKuKe1aR1I4pr1Eef4OjQMZKLWiXDAHTvw2SNEZBeNJSx7A3A508dD6n9aLSu+D9/EIpsXxr1lHweTiD+jwhD42M2+22mG76w6i9Z8u06qncRxVcDZRpjIKEfsVuReAORfpNFS/8W+/W/hOTI5MIas3fStIjPaSharqzE5f0CH0T0g4h/UNo+p9NG9QOi9gF3W3c6FJ17FGxSvJYSLnbzy3MnRpukpaqI/7Xasceq1evG4yIvumh3uviCC3YiPCAhGqG4PXMV1k1hIHO7HogmhDMB4KYhOu6SbQr0fimOXzherRwd/cbDJw6JN+7DssdEI9zb46QwdwZClg20r/Mz3qNDblPXrZbJPVE2dLBaPToK3x95fWXom5h/yt1TL9TUNptqZMgrZjNbuap9dHRkJPoTJ/tdYK+GWIubfeI5NhklmbpZn3t2q0rPPSkL3ghAb/uuzZNonoupB7sbjldh5ESlcnQUjh5Q5L+CPENbFXvH86ElLDUdW6caX+JmOm4eaaq41tiRxvqnN13ZZI5JEat5/DCBexxLc2bbJMrVzfpBBtzTWq5mA1DYFcNSiBZX8pU71Sxbi2XL3QxcwN3cyRMn3Ey1NKAlXdOkO8p8qbstd2tZs91NPfUdUDsx1ck3C5ypCJO4cv93yki4nLS+vAinOU4WHodKEaeZaDOPmedX78PZQVTKGZzZhsK5MzM8HSUdO0ha309aP0BaP0jWOIGIUe6NCAFCWM28+R/B5HMsfnbdxFqStOIan/+fX6KR3oll7ydLdxL1KFFJMQNPe0nTDcTzPkKJTWzad3F+bMtkMdFJMytPdfHMFXMgSorIqED+cUZo+0xoU7RpfSb9PuowKh3X3v7hYrKKXbzv64peJyrz80IWkjNJF3PLhh17II+N22btQc4PPLA7bbhvxX1IhOYDhLtoljV6Bb8cvJ/2cnCOiahmWX3Ig26tVr9br1aTwsaTWLX6vhMmfFk1dApk70uRPjWxKdIjmCg1cftiFA0drFQo+kvSJEksy6wqovtVWyFN7m6ImogOMkskSWK33PJ8bfsjd/1pGuQNZul/EtHdGnpG8WAgaev9InnxCnE1y2K37OJI40/Bomva+2wG0DuF9CiyY/vWux6qVpO0SX+lgp1/vu53T3eIaJ2mKNw80r2XNLrW8pTGCVCNMOVvH3voPUNF8HdxbP7/9q13PYbzpIQSTAjeFVWVsjsHRQPgzegzk1CanyKrxvcN4ToJIXYc1Qjwb6roweZS9OY+X+DSSmWccV+C+4LcOQOCpqLhmEn29Wrl+8OTVwSdHs2XPGcnQY6MDRDF16MaUeqBsZM7iE7sbDk/ig9AIinIA2SZkaVQ6lnOWHrD9J27FXRuh3Ataf3nSMd+lpPRzxHkZ2nUr4lUAr8AACAASURBVOXkS/8HIjuAlNEf9FMq3Uyp9//js/tvnVJkNxEjuT5l6JUHOLzyM8ThtaT1X6Y+9nlK8UE0GGZG/eR8gt5KpA+y6G2Xw8ZxJjnNu8QnqduT2y2IuYGnhtfBUnJ5tPPH2769rQ0pWNGWVPxUl3ASPefAf9SxSyNCfDWiJmBN+5yoIqqHTfwAdPbC+1jPQbf0cBFnaOMrO4orooOO9I+rn+MQBEZcs1pnlVYONetHTiyI45GgEaRtFq6m1wIDHcnwY3n17ok9RlGoC+SFSGWCGwiE0yrc25yHbzx858Ht1aGN4v4rno19VFQeEo0Oi2hK4RgaL3snglmmDstd+DCjcVSYGZjw2hJBjCPFSBPu48sue76myAtISPPzLc5B8nMQZRVu88enq/g2S8F9GtNOPoaITPrdEcFAyiqyF3dEirAmwRR6BVlRrWJr1xLltlyMgkE6uh2V/VLEznrWKLv5RbCkH8Al/KxoZDhWOHNURA+QsTe/dKeTauhn96wkYvREK/BsXe5gQlGG8f71fGbPGyd8Fu99I5959k14I8ZtBFFDxBC/iS27TnEfSUqqdY6uHeWui0Z438tP8K5XHuLoXzzO0OGP4GPvIEv/BNE6acOwdDUiG1my7JKOITxNafKOl9c48ud/g/a9i3r9DtLGnxLFJ9AI6jXQsJhS+WMs3bOqGZI0UcX2JuMZt8xPbY+jzSvj1BCpC1ITpCZyZh+EGlBDfHoJshN959SLPSFPPHZncOJdVgwucjzKQsfAb0isp+fQMHBMVWkvC+wO4tILEkNhMyzGbf2djjKvNfdoUz+104RMYbyGTX64kiTRRqTmkp9H03c/V2+gavWF3SLH/ou4v8fTsd8F+WNURmj6porxRFDPUhC9JoR0DWitKfw0YwUACFNfpM30wsyzurTJSs1XiLur4QvcPPY2ppFL9lkaEXUMiG97kRwZZw5FzwV6Ef8ndxsZZ+aOmmW94K+47JYl5YGBwWU4a1pFkQ1RnkD0ADC+sJ1GpeVZyJYmSaK4r83PurjOKlia7g2hdPA0pr5F55nGQTbVV/cKyCCWKY0xQ/RWouiPCD2fm/iJ/yj/lN6PWx9uSqMGGl/B96KVM4fYOJTHtPOyC9uMw2v2kcUfAdtCFEd5LCSXIvqOZsjYVPrb7J53Lh3lhVXbKcfvx+obCeEQGnImKXI5pu/gwgMxietEFRumMsJTqN2ipDmDo+ZCzdXqLlZ3L75ltm3qAjXwus2kBHSi7xxGII0/jrnEGkkeqNuyXTVvXJd6o6EdCysAVKuYIB0YqBgaVCZyiVlh5uq92Sn3mA06BsmfEZqmgSStVF44uGHDi19qjI1+yN3vEuFA4T0eH89xVKLY1K91UqWI5/TCwTPZMz89/cW3FDpsXso8br2AJrhL0jRk07zkmpCxcRW6SamBO+UU9uCyVzQycTcH3LNYkRXn/yCdLxGXiJb6MENENEsbdXWextLv5jZJDMHcWCoNX/zEE6v6EFbiha3U3VTDCGL/dGYLuZ3FszLOYPQNSGFL1qBEpQFgGSJLO390MSGKgNzuV4oW4375zI4agU5l9NvV96MrhsjsHiwbHY+Qc7uVe3f1zZgt01L/jRUHRvDz/gRr3IOEEUQhrZcpla9mNFsGc/AEpSmIWj2gGJh625uh+aKcZdudVHBcT9MGOUfPcLWKVSpphER9orlHeFzykkLddclVhZz28ZqGDr2lkk3jUUy0Urkwdk72NVlqy/nh6m41F6nLhBqJZ4hxlTLMvN8s0KJzbkX05hxVKsnw0MJlWwaODcVBo4+5Wb9IW9FVHHHWgMduTRUcaIsBPRXG59llvOakC3VEwFrsMZckJY4yZszbdbfzRbStXsr4CGnJ5TBBtnor9lFxjBAPYukCsNeqKJm4iUQK2d5K5ej+rdsu2Ccan3DL+t1dRWxQRFaMjIwckuCL3VtXwtyPoZxe9kzz/Jrc8UxtkPfuvRT8NWSN3K5kthfP9mAetdJrOw3tA2i4FKxMo94P0ev4+D99ie+fGMkXy/r26dHRYq5P80f7dhNK64qCFSuQsJIkyVMaT/UCuf76lOQRWPgzX6As/waXDQgpqsvRxjIS2TdRxT6ddMKNG4tDPBWRmkNNoO5IzZGaS/E5jTbqNReti4fTu4RzJEHmapSWaa7SKC0lU3Nj4xFROdQ+Ty0Hji2uYx09dEkCjdLIgIsvNjOgXfoUHDuheYXjlq3wNJhS59PPOM3whNPs/9Q4VQBztZqkg0d3W+S6WzU6RFtgeZ6P7gAxPiGb5bTombCvkJfTcx8SpD6+zEfBdTVEajbVeVOcSxF9wEpErKm+53lNggjHwWrm2T+4pXVENF9SRUxF+qGxGPe1ZllhRwSQJ5MkMXU9KKJDCCaCOl520VeGYKtVS3mWkGOiQS2r71Orn17udfPkzxYRNxKXI/KMpRouG3n+lb+Enn8bPaXpP0HuIpSeyV9KppTii+ntWwnbjLMNoHbJFwVzz71sQeaf4ohJqBiMHaFeP4Bqmj/O3otob37Krb9nhsjNTWuKmEEuR07Rfjrxu6nPjpF7XSU79xLkxLp/UKmgSZKk69dvWolk42EW446/nA8edOGo5OEhxc+Cu6mIDqpwCbBzciB1ksD6DaxRiRabp4wvN5BXuUnF0n2GRHqGrOicmmDPoP9OZdSa8zxRwk40l9qzMnh5siMwd1n5CYR+0dzHebr0tDQANHegaOruB1TCCcda0qKTB4wrVyVJ8qVOmkClcm+fua+T9vvZx42jB8BHXMMeNfYDa8wzlTy4e74RLhVhZV60Q3C31Mi+AZAGORwsPYSzGjBRAdFV7vYDFaWotI5IhEj69Wr1fSfOrIiwnNnNkiTKsn/fT+Pk68kaoAFE9yAndwDw/JJa5wML5jfwjv301J9Gw7p8jRlbidvFcN0cxDrnWWb5v2ago62c71nWg4t+2vAf1HKeZNY+SR1Y48RMjqntAm2MXyH1fGU6y4qU2BwtBaa1TSe1WxARyzNWbAYJshN9p4/JD0ClklCpJLr1Eb9LVPvNsjw+zwsmaKkiPEua7XMNI7j0uuQ5u7ntSGNxfxvwp8UImveLwoVRaiOvV2WBu1vTGC+CqZaGU8+eELefZ8JbY/bnNc0V4mwtKGf2LCVarS5a7mK3O/5MpXL/1mr1jmm88HDllQN9mcstkqYrEJ9EsIDotwS5zJuhQPlmbb+zZsbE2VEJqWm6C5FDIEvHexHUrAGU3vjwwwvur1SS/fnSxq2eTLhRJVpheXC7FhRansrOznovwyHzuro+jdvaptfZ3frEea2jA4ghqoAcDsiTAFHmQ+bZXtFSxTyFzFXUVpl5LJKNu/TMGmTIGdZXPxsv9kZo7LuEnvJqxk6ChgjsSYLlDq0Z6ywmyvFVIyx69h+Ie9/C2EvzcesnlK/ip1Z8gUsPjHB62eQth9GSvQO4ryJLc6btNkw9O3L65/eDXlwGsbQo2yajICMwOdVwfIXA5k0jrfY0T4umpRTSmqOWhzugrcfcaQmUxcbJAmZ72y0X1CSawYvdib7ZY+3aJB4cXHS1iS/1NN3nrieiKMRbt/pKUb9DVG81y3TcvuS5ucXhYObp0yX1Iy6lRxG/Ec8lcgTFUtMQ3bi+cu//1hjr+X96eg4VMWoLyyYnbw3S83bL0phchcpVJtHIspMHAjxs8PNeLHrkM7C8TpjgZsgdSLTbICevHHk6aB07OyRJYus33Ls60vPuzGxsmVntmfWVz2zH7B9V2Z8GhqJMLAvSGzJfaeLvwv1N7lY4UYq5QcnS2qiKPezwC+30nO55tJ+/4+oi+ywd+6ZoWGd56FbO7NxNlLUhkg/Coru3bHnhcJKQVqsXxnnNR/+ISRp5U5b1XMbVEO03sr+76crjI7t2ra0NHRv6Bwi34pTzQPJ0PrABsd7WlZKdwJE8E+aukfXXf/op1WjY0rQ/L4jhqwVZbtbIox60hFu2uyRHnzytk++E5vM203KsTSSee5Nl6XqcBagaGp2g0djG80PD8MDMYyWJkWxULNpO/eRhRPoRNczWMy9dyrZte1j0zkkHzeKhXvJ8GdffptSzgEbNiGIwHuPFVUdy73el5c2eaclZqkr2skvp6bmYRj1Pa/TsAMYhEtepSy6cUT1IrUsza2Py8ZM16RnahhgK0YTg3kk4i3qQuXTzU72m4VfE7TcJ0Ql1GTUhQhlAQtkss0lDGGAisr3k8QGIR8xH/0IlrMN1QdOp4DmTBJcPx3Hj1akt3HbttYxmLlep6O2epUvBtWlbaxaeyCz9XP1kOtRT1gjBcLS9HuRsMZVlZMW8hDNijNB8lGdPS5IkumULkWSsymx00N0jCdGlAusMUhOGg8mwo6mYlc19UDXEmRW1KNqcHqKKW/b5RoPDUezllg9b8NNw0sCkF4N7/gIJ/ldCuFHUV7lleYiNoG5ZJITbHR+8YHDwi1+r+rGgtVWWydtEdY2bjWsADiaqdcuyh+aVSzvzEKPd6QvbFz0j6BHwFYVwoUBuG3Mxx8zddo6OlIab8/a17faMWXZCkCKHXGKYGHcqKtXqI8k06uypZ2EqNkIyUzTARqCqLBlcisZXktbLedSF7CewO2dC15/aX5CIkTxygMVLHyOetzZP99OVqFxBkuxm0+3ka08V8OKZvo4iYHsjucpaqM6Lvr0Az94KelcRagRuJzC7H6rK4LLL0W/3k922k7suOjI1pKjoKxHj3r2XEOR3SRurwYxo3ijpS9tYYIcY6iRBTodpHDgaxtLM4xqSV0M5mzx4AcMhUzk9G+RpPC31uBzHKQs89zAOoDIghSrtZHnwdrPb3GZlInoos/pfBV48AZDFi/5eG/yChNJveFYvN1W+/CR8vov8RkDfCpK6WX9epqrlnRUXE1V1S78QGPt8Z4/zGbpG5Ix9lB26On0MDv5Ur6Gvxr0XUMtSy/3FROLaj0o/4uNOmMzSybdWKqqK2ZMe/F5ixnn9mUnAHc6jAcdeHHx84cKhTaLh4+QRNCYi6oJC1gv6JhWtAKPu3gfEZqZ5EXsHxDSUEOdxs9q9Dz74nuMA1eojkbL7oIscQFg5ZXwRUwnHzPyfb7nl+RrkNuqr3pDuK9X0gGi0sjBUNZlwbj7FasC2fP8zWXvHARRLI5yL2LT3ZngO/Fe1df81K+Y3289C9DLDWIPIxUVoD2SN3YTy1NUBZ0Jyfcpn9j6IZe/GHUKIsfQm4E8mO+EQYsT72D04zIW/njK6OyJ6Wxn2LiCTdZTC67HoTbgtAIworuPp54nqW7lwRR+mb0PCrdT9m2za8yD+rd2kpUMMMMxL56WE28qk+xZz395LifRdIFdjmVEqK86TpKUt7H5FSlIwtdmZqjo/sHWLLcJriMbkthhMMHVTkyh32bppvq1gPqKFimJKsX+zPwXIZggU74RZPjdJkthrX7u5TMziwnsMnqdw5fbrdkkjV/5D6BnNvPG5gD7ctpzB0A03fOIPGo3yAo3i2y2tNyWaXDV3U3fpQ9wQz+v3FZKPoIiqmttXAvLhavX7w5XKwl6bUUL/yUA+v5+YX4rDxS5mZm0vnPwFpLl0MEntzf/Ns0tCrJ6lzxD8w4svGHzm8IkXFnQebXbocGtYCKndfvvu9IknBv7kpZPyStHwW+T1N1NBiqfBcJMyeWFammuku+dZPSGU1PG9Da+//xtfP76nybSq1W122WVLDp/Xlz4jGq5xyyLaXroI6iIHVdnfnDOAN1yVnPhadeGOoGFDXui3FWCV2yzZL954uv2Y00I+x0paLxNKt1OK3zTrl3CWlUkb/eBQikcYe+kJDi87cdqLcIlvJ02PoNFg7qxhPZv2DY4vP49ofhvI5YSwGWSYWqNOiCKM+USlBZRKg2SNATzLmWpcTmmMfYGGf5yja0+waM9yovJrEF+KyFuJz9uAZ8fRxnFG/BiM1ElLfYQwSFxaSv1kwWR7FPchxkY/xNE1+5vnNlHgG1dX2yeu2e7MhcolTOCkZz7q4qPuPiomNXcZFfOamNda2/Lf3bzmxfb8t3w/cR91l9FsxjjITvTNHqVSvdexQciZFS4mxSdPe5O0CKlINcRDDat/eNEFA/8lL4TQujGvuebEIZEjv25p/ZOi4VirTmOzVqNT2NVM0BTHVCOTEB9yz/6vQPquavU9z7Q7AYq0RcPF2p+pjkGzraMoDMtN+ovtgbT15kvHf5dgrRTCTjjJeICqF7RIUQl4Fo9DVupRkFS1NKIarIitMRFJBTWcPG3O1fJ2HjKjoZRq6DnmWf2PLbLbtq8/+vBFF+1uuw/yfvL9i3Oc1eOpNK9JM60xyyIFuPLK4yPnzcs+hGXvFaI9QeNiPClSIL2Nkef0qqppKJ2wrLElqzdu+Ub1xR2txcEAEnvqqedruD2hWjohzb5a18c8G9sD9XEJrOn1D/A1MwMN7fsX9gd/cmysMTQ5rXLWEPL7BAHL+qifXEy9NrtPkzlqgLQxhPmjpx2ek7hy56uOoeEhQpQ7Yks9g3h6I9Rb9ImmqPQTQoWo52ZKpbcQ4lsJ0QbMLqZRGwSUuHcUZD+1l95Pze7k6CtypqZaJkQpUZybIhq1ftJ0JSJXEKI3EUpvRsONWHYJjbEBRCGeN4LZwzTGfpGjax5vJ7tDPcjJjHBm8axu5BWfFdP8T4H266gdtnVoN3OwZ7JBdqLvtKSvKBL0sKiWTaQPtzJ54QkDqSMyjPsQlu0Usb94tPrbDwM8MMkWXTwQtUrl/g+kfvKL6nabhJ5LgWW49UlegFVB6yI6jNgRS9OnTep/dnxo0WO33747bYZqnH9+ZN//QXZYNX7aMFQL35UEGo2TB0qlUsfsjgaMlDXeIRN0VDFERyRNR4AR1Z4draI2CrghOuI6Ntxxek6GNJSj/aj0mQYTXB1MpaSucqjt3Dvi8eoLB6+5ZvBOVasgvFajaK0QBtyZD152L7SWfC2WuiDH3bMhz+o7UR5UOfbQhmuxR5PEEhK9+sYoVQ0HBN1pmk2gJ5NakW43MaQqSUA0OhZC/DRCLG03mkjpsPjJ0eYSq0mSjFSrfLbuCx8LJreFKGxwD0vzXG0rjpVUJIwAx9zGnvEs+++qjYe2P/q+E52X+YVqlR0i4fEQlZY1tzuYalxv1EYeqX69FarTCpy/d6e7PR6intjVinPNXyBpdvJrPT3DwzOVmpsWlg0T9T4DVj4jI5ijBUNTRr/3GPN69p7u2i7jCPwVIaxFepSe82Cs9mpMHqdU3oPQh3kZiPHm85NnF0GooTJKo3GcNN2PNZ5ArMp7Xr13Qmrh86v3snTPHWR6IyLXEc9bBT6AWR9mEZiimiLRKBKOU39pH7XRv0PCF3jPq4YmO67yJ+uze2+g1LuZdGw5WTadwp3r6I3aX/Kq//W2ZFvFkkTs4986uQLxN6vPQV5b4eixzKvvW3teHmN1775V9ER/i9uaYvW0Dge6EfVAlj3N83922UwXr1K5v5yFk6s9s+UqMmDIAnWPwVLxMOyeHVHVg8C+SuXo6GzVmZtu+uT8kZFohUS+SmCxYX3iquJ+3NWPqLf6hElMJkn0tV/tX1YqlQbaOWFQVxdGouzY/k6LTV150yfnxyO6KgstVScGsiAWsrGDJ08Gi+Ppf69W33dicp+33bYlfv740Apx+jJrHRfU1cZKx77xjTtPmQPcZBqVyr19WQjLQ9YYNNEBy7yfQF4d3RkVYVjdh0APQe+havWOGsWSuW3ZNhEsXJGpz59MTzAZrlbv2teJhqtv3DQY123p1DeLpmPn6/6nvnjnuFzelOB27VobHTl+fJVYusKdpYL3g0YOI2I+BHJo3ryePQ8++JvHTzUHt922JT569IWVmUpvO90A3jN28B8e/A8d+kj06spPrw1ZiJvX7FTXa1b4410D1MMymqnFTWGoUXzP1G7/PxJljCF+75WHzogOgHt39SHzVhIKPpPKML3hEA1bTqO+gCjqwzxGPcI9ArW8iogWoTc+hDeGOLo2v36d1PymY2fZoX7Sl1biuhjxAdA+3CPUR3E5TqZH0Jf28Z6fG5qO3JzbbNqzgZ6+zaS1FTmX7Yj8DdKo/w090duS766oJ4nYJ58bXeaZ3+yEGMfOyktjBqpIJtX3ru3J04U2P7sGjf8WfNW0DNLdKPWAZzt41yt+YeoOE9G+/nG+ZOtLOjT0Xbv9dtL2dZFP19bTYgxJBBcW8/jdZimufK3safucSXWa/phKBW0vedUsk9XcNt3veYzf6fU78zEdeimqgrevTz15/NYa3zP1e/r05BELE49p+3WasI8Wc06SRHftIjp69EJtv4ZF37Ocg6nX9NTzOPGY2V2vU5Exi3VgZoWqwjY7Y+lxCj3NcJxpajlOe9wM+0zYv2CUrf4Vqkwc8+4ZUxJzbrP52Wso9W6mMbYan4FBaqRY+ijiv8Tzq4+TiG1+1hec9Nobxa0X1bP0oBpmmhJk+/f//P88kCSJsenZKwjRF4EFZOn0EmRpHmTpdt698vrZj9fK8ICm6jIXC4ZN7vfHbRGyHxXaM2pgbub63GFittWPN61dzAKniovsACFxZelzl1Cat5n62OXj3qGOfhkB1b1kY7/MC6/eTSJ27y7vS8NL17iEQU5Zx/HUUPfR1OZVhx/gRJKIsXnv2xG9H/N4gkNmAn1uxL2QNv6ad6+8bVYBsF100UUXp0CzWMUwaTact8fTuXJMKExrRqmnHymtgbtJ3PXoEDVTjoh7TfC647Uz/Yh4aipDw0O0ORDCL6AhHndZji9X10afA5aBUtjHZrn+bhdddNHFDMgZZNw4QTZ2pChZNFHymqzSZul84Cou/PU4AZLrJY0bHBHXE47XBK1LpnWh7XPKttcFr5tRH3Pbz7a7cxru/04ZYUPhYe6cqSPFtiyFzJ6d+ynqoosu/rUiZ5CH1p7A2UUUj+YS2jRhMyJKlsbEPeupp2uboVBHh847JioH1b2mntZUqam3fU7ZDjXB63h04OSreo/AxrwOx8n6G9FwMWld8WncP05RXUSOIeSOnblcg7aLLrr4V4vWUonC0+CdY+Pa4Q5ZuhbRm1m4u5ck0eR6SV+M4wOWlo5khLq518y9ZqH4tP/f3m7bniHHYi/tTUQsgTzfslS6sxhzyuJTEyGgYTcuh7r2xy666GKu0JLKgj5NOnaIEGkH70wbXHEvA/8WDVfkbnTX5OVSmzcW71NPjyleV3wio/S2Txtz1NTrkqbH5WR939G1jJK4suSpMpK9EwmvIa3TvnznFIgYuGHZDsbsBFw3RyENXXTRxb92FG5vMf7XoSNktpWoB5gpk4XcIQIr///27ifEruoO4Pj3d869972ZvsQYnTCRYEIYUpmFRBoGXdVAd13ZVpe1QWiKWVYLUkrvUIrYLooUq6YuFARtCy5aKaWbDLRKrS66KLY0dkwlZpKZMB3j+ObNfef+jov73sub/2/GSSPl94FhOMx973Bn8eOce3/n98P5H7L/vapgZR7d6RPS/O++xrRGuaROm1LGIJIUErQQ6fsJWlR/06IUuVxvNqY/Or7vWt7dGWvjXlz2CGW7AVvkcImAS66i5RvMjy2Sn7zpLWONMf8fVi4Vf/HPu3H+LYQM7ZSFiquu7tWHFCWtKaF4lVA8ztzs1W4CZh6jOzhDPSx/spdm0mg5XHSFYxnqaaaFoknQlk+GFubGaeYiSn4ugfuVQ++fILpniXo3ZTtZVeVj1ePRCN4r4v9AaJ3hyl0fbPsAvTHGbGDtXvr5f7+C9w91muC4zXfbUcnqBWX7t8TiKW6Nf+fd8dAfpPJzMeEIyUhzLoER5marPtj5SQnXM+MnYeTBYZyfIKs/g8a7KNsbTLpq/trwAq3mE8wee2GrrHhjjNmO6+Gv+3Lj7L++giQvEXWUUjcPkFW2tuLTgJbvoPpL2vIa82OLOZOdjhAb5CT2H/85cP5OvDyE84+AHKVsb/0cMaIkCSBTEB7mw7FLtno0xuymleEvzx2HH95LO/wY5Nuods4vbkkRgbQ2S2vpjzh+Ra35JqfuWVj3HGg3kD3z/ii++Bo++zqRE8Sy0TvJM8iczjtUH+Ty2GsrvtcYY3bB2kiUR8fBfxwn3fNzQjGBbljdp09nJQmQZAqySFieBvkLTt6mHS+RyiKxdJRxP94fBb5EZILa0CHay/XqxU/cOjjG7vPPuqLlr/mweQpWbuuNMWY3rB8gc1GeO/8NstrPCMVoFSQHLNsdY7Wa9KnDewgBNFR9dKvVaB2fgnMQ2lAG3TSNZ+0EikuA+FdieYqZV3Zem84YYzax/vY3jw75wu9pffIsiEOcDlyUVsQRoyMUyvKSom065wHrIBkxQnsZlpd08ODYPd0TOw165AKqP2UmTG/jXo0xZls2Xhbm0XHLhb0Mhadx8k1Uldh5ntjrM9qp5r3huG+K6+lBdBqUDPD5vjFU5eLTbJ6y/AHt1svMjTdta22MuVE2Xr3lonx05Bqe76O8iEsCzmkv6PWauMsm41U5jL1CE4N+vvsVUq0c01qL0H6C1L3I3G8sOBpjbqitHyzm0THy7gF88jhJ7Vto2IeuetPcW+XJjRgr3iuRi8T4JKfHzu74bo0xZhu2fv6XizI3PovwJGUxSZJdxGdVWbQYtfNWmV7zrN0aRxSRquct7k20/C4Mv3xD/xvGGNNnsLfHuSgzx+bJ0rOE9hkiUyRZwCeuU0OyIn1b452Pq+CbZHRSh14gLJ1hf/t1Zg62dnSXxhizA37gK6cmI/fcqnz8wHka8+dQvQJ6lNrQHlQFYlldGGVNy4beKrFroz7bUqXwJGmLMryDxu8RWs8xO36JuRG1Z47GmP+lwQMkwNRU5H4RFh+4xmO3vcFXH/0dZXsJn9ZIa/Wqx7QH5yIinf1ylPWDo4A4xbkqenrfojZ0haL1JzT8BIk/4jvH3mbiQCA/qUxNbqf5tTHGfGYDZn+vo9eshxRnXwAAALtJREFU+8uOO0aPojIBch/p8HGkPEQobyfGYbzXNdNEdagqIk18chHVC4Tib0TewvNnTn/xam8OSwI3xtwkOw+QcD2Adc9b73+vQcYhXLyDUu9E/GHSZBTxDaJmAGhs4uICoZyB+AGlTEOcxV+7zMzrrV4fW2OMuck+W4Bcrb8Rd34u4fCRhI9Dxp7EsdC5xgfFF8rwcOA/RwK5hF4tSAuMxpjPkd0NkP16W3BYWfJssjPu/LagaIz5nPoUBSp4D1AF9yMAAAAASUVORK5CYII=)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Factuality_Test.ipynb)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**LangTest** is an open-source python library designed to help developers deliver safe and effective Natural Language Processing (NLP) models. Whether you are using **John Snow Labs, Hugging Face, Spacy** models or **OpenAI, Cohere, AI21, Hugging Face Inference API and Azure-OpenAI** based LLMs, it has got you covered. You can test any Named Entity Recognition (NER), Text Classification model using the library. We also support testing LLMS for Question-Answering and Summarization tasks on benchmark datasets. The library supports 50+ out of the box tests. These tests fall into robustness, accuracy, bias, representation, toxicity and fairness test categories.\n",
+    "\n",
+    "Metrics are calculated by comparing the model's extractions in the original list of sentences against the extractions carried out in the noisy list of sentences. The original annotated labels are not used at any point, we are simply comparing the model against itself in a 2 settings."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Getting started with LangTest\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install \"langtest[langchain,openai,transformers]\" "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Harness and Its Parameters\n",
+    "\n",
+    "The Harness class is a testing class for Natural Language Processing (NLP) models. It evaluates the performance of a NLP model on a given task using test data and generates a report with test results.Harness can be imported from the LangTest library in the following way."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langtest import Harness"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It imports the Harness class from within the module, that is designed to provide a blueprint or framework for conducting NLP testing, and that instances of the Harness class can be customized or configured for different testing scenarios or environments.\n",
+    "\n",
+    "Here is a list of the different parameters that can be passed to the Harness function:\n",
+    "\n",
+    "<br/>\n",
+    "\n",
+    "\n",
+    "| Parameter  | Description |  \n",
+    "| - | - | \n",
+    "|**task**     |Task for which the model is to be evaluated (ex: disinformation-test)|\n",
+    "| **model**     | Specifies the model(s) to be evaluated. This parameter can be provided as either a dictionary or a list of dictionaries. Each dictionary should contain the following keys: <ul><li>model (mandatory): \tPipelineModel or path to a saved model or pretrained pipeline/model from hub.</li><li>hub (mandatory): Hub (library) to use in back-end for loading model from public models hub or from path (ex: openai, azure-openai, ai21, cohere etc.)</li></ul>|\n",
+    "| **data**      | The data to be used for evaluation. A dictionary providing flexibility and options for data sources. It should include the following keys: <ul><li>data_source (mandatory): The source of the data.</li><li>subset (optional): The subset of the data.</li><li>feature_column (optional): The column containing the features.</li><li>target_column (optional): The column containing the target labels.</li><li>split (optional): The data split to be used.</li><li>source (optional): Set to 'huggingface' when loading Hugging Face dataset.</li></ul> |\n",
+    "| **config**    | Configuration for the tests to be performed, specified in the form of a YAML file. |\n",
+    "\n",
+    "<br/>\n",
+    "<br/>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Factuality Test\n",
+    "\n",
+    "The Factuality Test is designed to evaluate the ability of language models (LLMs) to determine the factuality of statements within summaries. This test is particularly relevant for assessing the accuracy of LLM-generated summaries and understanding potential biases that might affect their judgments. Below, we break down the test step by step:\n",
+    "\n",
+    "#### Test Objective\n",
+    "\n",
+    "The primary goal of the Factuality Test is to assess how well LLMs can identify the factual accuracy of summary sentences. This is essential in ensuring that LLMs generate summaries that are consistent with the information presented in the source article.\n",
+    "\n",
+    "#### Data Source\n",
+    "\n",
+    "For this test, we utilize the Factual-Summary-Pairs dataset, which is sourced from the following GitHub repository: [Factual-Summary-Pairs Dataset](https://github.com/anyscale/factuality-eval/tree/main).\n",
+    "\n",
+    "#### Methodology\n",
+    "\n",
+    "Our test methodology draws inspiration from a reference article titled [\"LLAMA-2 is about as factually accurate as GPT-4 for summaries and is 30x cheaper\"](https://www.anyscale.com/blog/llama-2-is-about-as-factually-accurate-as-gpt-4-for-summaries-and-is-30x-cheaper).\n",
+    "\n",
+    "#### Test Implementation\n",
+    "\n",
+    "In our implementation of the Factuality Test, we instruct the LLMs with a specific prompt designed to encourage unbiased factuality assessments. This prompt is structured as follows:\n",
+    "\n",
+    "```python\n",
+    "\"factuality\": \"\"\"Decide which of the following summary is more consistent with the article sentence. \n",
+    "\n",
+    "Note that consistency means all information in the summary is supported by the article.\n",
+    "\n",
+    "Do not bias towards the first summary.\n",
+    "\n",
+    "Do not bias towards the second summary. \n",
+    "\n",
+    "Both answers are equally probable.\n",
+    "\n",
+    "Article Sentence: {article_sentence}\n",
+    "Summary A: {option_a}\n",
+    "Summary B: {option_b}\n",
+    "Answer (A or B):\"\"\",\n",
+    "```\n",
+    "\n",
+    "#### Data Format\n",
+    "\n",
+    "The test data is organized into various columns for clarity:\n",
+    "\n",
+    "- `category`: Specifies the test category as \"factuality.\"\n",
+    "- `test_type`: Indicates \"order_bias\" as the specific test type.\n",
+    "- `article_sentence`: Contains the article's sentence.\n",
+    "- `correct_sentence`: Represents a factually correct summary.\n",
+    "- `incorrect_sentence`: Represents a factually incorrect summary.\n",
+    "- `result`: Stores the output when the correct summary is presented first.\n",
+    "- `swapped_result`: Stores the output when the incorrect summary is presented first.\n",
+    "- `pass`: Indicates whether the test case passed (True) or failed (False).\n",
+    "\n",
+    "#### Evaluation Approach\n",
+    "\n",
+    "During the evaluation process, it's important to note that we initially treat the `correct_sentence` as A and the `incorrect_sentence` as B when calculating the `result`. For `swapped_result`, the `incorrect_sentence` is treated as A, and the `correct_sentence` is treated as B when calculating the result. This ensures consistency and fairness in assessing factuality.\n",
+    "\n",
+    "Our evaluation approach involves several steps:\n",
+    "\n",
+    "- Bias occurs when both the \"result\" and \"swapped_result\" are A. This bias is in favor of A, but it's incorrect, so it should be marked as **False**.\n",
+    "- Bias occurs when both the \"result\" and \"swapped_result\" are B. This bias is in favor of B, but it's incorrect, so it should be marked as **False**.\n",
+    "- When \"result\" is B and \"swapped_result\" is A, there is no bias. However, this statement is incorrect, so it should be marked as **False**.\n",
+    "- When \"result\" is A and \"swapped_result\" is B, there is no bias. This statement is correct, so it should be marked as **True**.\n",
+    "\n",
+    "- In cases where neither the `result` nor the `swapped_result` is \"A\" or \"B,\" we employ regular expressions (regex) to identify responses. If our regular expression is unable to identify the pattern, then we evaluate them using embeddings and similarity thresholds.\n",
+    "\n",
+    "This comprehensive evaluation approach ensures accurate assessment of factuality and helps identify and mitigate biases in LLM-generated summaries."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Openai Model Testing For Factuality Test\n",
+    "\n",
+    "In this section, we dive into testing of OpenAI models in factuality-test.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set environment for OpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "import openai\n",
+    "\n",
+    "os.environ[\"OPENAI_API_KEY\"] = \"<YOUR_API_KEY>\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Test Configuration : \n",
+      " {\n",
+      " \"tests\": {\n",
+      "  \"defaults\": {\n",
+      "   \"min_pass_rate\": 0.8\n",
+      "  },\n",
+      "  \"factuality\": {\n",
+      "   \"order_bias\": {\n",
+      "    \"min_pass_rate\": 0.7\n",
+      "   }\n",
+      "  }\n",
+      " }\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "model = {\"model\": \"text-davinci-003\", \"hub\":\"openai\"}\n",
+    "\n",
+    "data = {\"data_source\": \"Factual-Summary-Pairs\"}\n",
+    "\n",
+    "harness = Harness(task=\"factuality-test\", model=model, data=data)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'tests': {'defaults': {'min_pass_rate': 1.0},\n",
+       "  'factuality': {'order_bias': {'min_pass_rate': 0.7}}}}"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "harness.configure({\n",
+    "    'tests': {\n",
+    "      'defaults':{\n",
+    "        'min_pass_rate': 1.0,\n",
+    "      },\n",
+    "      'factuality':{\n",
+    "        'order_bias': {'min_pass_rate': 0.70},\n",
+    "      }\n",
+    "    }\n",
+    "})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "harness.data=harness.data[:50]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Generating the test cases."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Generating testcases...: 100%|██████████| 1/1 [00:00<?, ?it/s]\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": []
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "harness.generate()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "harness.generate() method automatically generates the test cases (based on the provided configuration)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Running the tests"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Running testcases... : 100%|██████████| 50/50 [01:05<00:00,  1.30s/it]\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": []
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "harness.run()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Called after harness.generate() and is to used to run all the tests.  Returns a pass/fail flag for each test."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Generated Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = harness.generated_results()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>category</th>\n",
+       "      <th>test_type</th>\n",
+       "      <th>article_sentence</th>\n",
+       "      <th>correct_sentence</th>\n",
+       "      <th>incorrect_sentence</th>\n",
+       "      <th>result</th>\n",
+       "      <th>swapped_result</th>\n",
+       "      <th>pass</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>the abc have reported that those who receive centrelink payments made up half of radio rental's income last year.</td>\n",
+       "      <td>those who receive centrelink payments made up half of radio rental's income last year.</td>\n",
+       "      <td>the abc have reported that those who receive centrelink payments made up radio rental's income last year.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>five ambitious clubs are locked in a scramble for two champions league places behind chelsea and manchester city and it could prove more tense and exciting than the barclays premier league title race itself.</td>\n",
+       "      <td>five ambitious clubs are locked in a scramble for two champions league places.</td>\n",
+       "      <td>five ambitious clubs are locked in a bid for two champions league places.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>but it wasn't until last year that the 25 year old from pennsylvania went on to become a global superstar on a level with the likes of beyonce or adele.</td>\n",
+       "      <td>the 25 year old from pennsylvania went on a level with the likes of beyonce or adele.</td>\n",
+       "      <td>the 25 year old from pennsylvania went viral on a level with the likes of beyonce or adele.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>seven games involving nimes were investigated after conrad was arrested last november.</td>\n",
+       "      <td>seven games involving nimes were investigated after conrad was arrested.</td>\n",
+       "      <td>seven games involving nimes were arrested last november.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>the driver's side of the windscreen immediately shatters and falls through.</td>\n",
+       "      <td>driver's side of the windscreen immediately shatters and falls.</td>\n",
+       "      <td>driver's side were immediately shatters and falls through.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>A</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>shafqat hussain, 24, was scheduled to be hanged on thursday, but a government official confirmed wednesday night with a tweet that hussain will not be hanged and that his case will be reviewed for another 72 hours.</td>\n",
+       "      <td>shafqat hussain, 24, was scheduled to be hanged thursday.</td>\n",
+       "      <td>shafqat hussain, 24, was scheduled to be hanged on wednesday.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>' referring to the ump's electoral alliance with the centrist union of democrats and independents (udi), mr sarkozy said: 'never in the history of the fifth republic has our political family won so many departments.</td>\n",
+       "      <td>sarkozy's electoral alliance has won the departmental elections.</td>\n",
+       "      <td>sarkozy's electoral alliance has won departmental elections.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>A</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>7</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>insiders say the row brought simmering tensions between the starkly contrasting pair -- both rivals for miliband's ear -- to a head.</td>\n",
+       "      <td>insiders say the row brought tensions between the contrasting pair.</td>\n",
+       "      <td>insiders say the row brought simmering tensions between miliband's ear.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>8</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>katarzyne pietrzycka, 42, was attacked with a stiletto by a thai transvestite jealous over her husband, leaving her bleeding heavily from her head (right) following the assault in the east coast thai city of pattaya.</td>\n",
+       "      <td>katarzyne pietrzycka, 42, was attacked with a stiletto by thai transvestite jealous.</td>\n",
+       "      <td>katarzyne pietrzycka, 42, was attacked with a stiletto by a thai court.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>9</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>the man, named only as saleh, said the masked militant who appears in several beheading videos was a senior figure in the extremist organisation responsible for murdering foreign captives.</td>\n",
+       "      <td>the man, named only as saleh, said the masked militant appears in several beheading videos.</td>\n",
+       "      <td>the man, named only as saleh, appears in several beheading videos.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>10</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>the bolton boxer had hoped to take on floyd mayweather but the pound-for-pound king will instead meet manny pacquaio in a $ 300million mega-fight on may 2.</td>\n",
+       "      <td>the boxer will instead meet manny pacquaio in a $ 300million mega-fight.</td>\n",
+       "      <td>the bolton boxer had hoped to take the floyd mayweather in a $ 300million mega-fight.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>11</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>the home which was built for former australian prime minister malcolm fraser and his wife tamie has been opened for inspection just a day after his sudden passing.</td>\n",
+       "      <td>the home was built for former australian prime minister malcolm fraser.</td>\n",
+       "      <td>the home was built for inspection just a day after his sudden passing.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>12</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>undeterred by the fact that he was unable to use his hands and lying next to several police officers, the man struggles to his feet while they are tackling two people nearby and sprints off through a watching crowd.</td>\n",
+       "      <td>he was unable to use his hands and lying next to several police officers.</td>\n",
+       "      <td>the man struggles to use his hands and lying next to police officers.</td>\n",
+       "      <td>B</td>\n",
+       "      <td>B</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>13</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>(cnn) the man who attacked a security area at the new orleans airport with a machete and wasp spray also had a bag of molotov cocktails and a car containing smoke bombs and gas cylinders, authorities said.</td>\n",
+       "      <td>the man who attacked a security area at the new orleans airport was with a machete and a car containing smoke bombs.</td>\n",
+       "      <td>the man who attacked a security area at the new orleans airport with a machete and a car containing smoke bombs.</td>\n",
+       "      <td>B</td>\n",
+       "      <td>B</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>14</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>former manchester city boss roberto mancini has turned up the heat on successor manuel pellegrini by claiming the chilean should be winning a trophy every year.</td>\n",
+       "      <td>roberto mancini has turned up the heat on manuel pellegrini's successor.</td>\n",
+       "      <td>roberto mancini has turned up the heat for manuel pellegrini's successor.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>15</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>but critics say they do not go far enough and will allow lenders to shut hundreds of branches across the country -- even when there is no other bank in the community.</td>\n",
+       "      <td>critics say they do not go far enough and will allow lenders to shut hundreds.</td>\n",
+       "      <td>critics say they do not go far enough to shut hundreds of branches.</td>\n",
+       "      <td>B</td>\n",
+       "      <td>B</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>16</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>and a prized silver dollar each could fetch $ 10 million or more.</td>\n",
+       "      <td>a prized silver dollar each could fetch $ 10 million.</td>\n",
+       "      <td>a prized silver dollar each could fetch $ 10 or more.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>17</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>the toffees were the country's last hope on the continent before a 5-2 loss to dynamo kiev in europe on thursday night meant they followed chelsea, arsenal and manchester city in crashing out at the last 16 stage.</td>\n",
+       "      <td>the toffees were the country's last hope on thursday night.</td>\n",
+       "      <td>the toffees were last 16 in a 5-2 loss to dynamo kiev.</td>\n",
+       "      <td>B</td>\n",
+       "      <td>A</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>18</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>louis van gaal said that he believed wenger's estimation of the necessary points total.</td>\n",
+       "      <td>louis van gaal says he believed wenger's estimation of the necessary points.</td>\n",
+       "      <td>louis van gaal says wenger's estimation of the necessary points total.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>A</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>19</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>visitors to the isle of lewis will be hoping the clouds stay away on friday for the solar eclipse.</td>\n",
+       "      <td>visitors to the isle of lewis will be hoping the clouds stay away on friday.</td>\n",
+       "      <td>visitors to the isle of lewis will be hoping to stay away on friday.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>20</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>despite their attempts to drown out the sounds, they claim customers are being driven away by the noises, which start around lunchtime.</td>\n",
+       "      <td>they claim customers are being driven away by the noises.</td>\n",
+       "      <td>they claim they are being driven away by the noises.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>21</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>captured at 1,500 frames per second, the video shows the chameleon attacking a cricket with remarkable speed and skill.</td>\n",
+       "      <td>captured at 1,500 frames per second.</td>\n",
+       "      <td>captured by 1,500 frames per second.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>22</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>mrs jarvis, who lives in the nearby village of farnham royal, said most of freya's friends have been allocated places at a school in the town of beaconsfield.</td>\n",
+       "      <td>she said freya's friends have been allocated places at a school in beaconsfield.</td>\n",
+       "      <td>mrs jarvis said most of her friends have been allocated places at a school.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>23</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>despite missing chances against scotland, for the first time in a while our strike runners consistently found clean air to run in.</td>\n",
+       "      <td>despite missing chances, runners consistently found clean air to run in.</td>\n",
+       "      <td>despite missing chances against scotland, for first time in a year.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>24</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>but now conditions on the farms that supply kfc with 23million chickens a year can be shown for the first time.</td>\n",
+       "      <td>but now conditions on the farms can be shown for the first time.</td>\n",
+       "      <td>kfc with 23million chickens a year can be shown for the first time.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>25</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>both on screen and on the red carpet, she always radiates with a natural beauty - but while cate has avoided going under the knife or being jabbed with needles, she still relies on some beauty secrets to keep her looking her best.</td>\n",
+       "      <td>she always radiates with a natural beauty, but cate is looking her best.</td>\n",
+       "      <td>she always radiates with needles, but cate has avoided going under the knife.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>26</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>she and her husband were trying to walk away.</td>\n",
+       "      <td>she and husband were trying to walk away.</td>\n",
+       "      <td>she and her wife were trying to walk away.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>27</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>his former teacher, 35-year-old brianne altice, has been charged with 14 felonies - including rape, sodomy and sex abuse - for allegedly having sexual relationships with three different students.</td>\n",
+       "      <td>brianne altice, 35-year-old brianne, has been charged with rape, sodomy and sex abuse.</td>\n",
+       "      <td>brianne altice, 21, has been charged with rape, sodomy and sex abuse.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>28</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>fans displayed the messages on bloomfield road ahead of their 3-0 defeat by charlton athletic.</td>\n",
+       "      <td>fans displayed the messages on bloomfield road ahead of their 3-0 defeat.</td>\n",
+       "      <td>fans displayed the messages on bloomfield road ahead of their 3-0 win.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>29</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>we reported that, just before we battened down the hatches, all the three and four-year-old boys had been snapping at their pyjama elastic and comparing the size of their willies.</td>\n",
+       "      <td>the three and four-year-old boys were snapping at their pyjama elastic.</td>\n",
+       "      <td>the three and boys had been snapping at their pyjama elastic.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>30</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>england coach mike catt wants england to be more ruthless when it comes to scoring points.</td>\n",
+       "      <td>coach mike catt wants england to be more ruthless.</td>\n",
+       "      <td>mike catt wants england to be more ruthless when it comes.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>31</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>gomis, playing his second match since collapsing at white hart lane, struck three minutes from time at villa park to prick the bubble blown by tim sherwood.</td>\n",
+       "      <td>gomis struck three minutes from time at villa park.</td>\n",
+       "      <td>gomis scored three goals since collapsing at villa park.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>32</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>just load all the luggage you like in the boot and head to the coast, drive aboard the ferry, then start your holiday in relaxing surroundings, with great food, shopping and entertainment on board.</td>\n",
+       "      <td>the luggage you like in the boot and head to the coast, drive aboard the ferry.</td>\n",
+       "      <td>the ferry, then start your holiday in the boot and head to the coast.</td>\n",
+       "      <td>B</td>\n",
+       "      <td>B</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>33</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>herrera also revealed that captain wayne rooney is the joker inside united's dressing room, but admitted that he has to keep the conversation away from boxing, for fears rooney might hit him.</td>\n",
+       "      <td>herrera also revealed that rooney is the joker inside united's dressing room.</td>\n",
+       "      <td>herrera also revealed that rooney has to keep the conversation away from boxing.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>34</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>to this day, the fame is untainted and the talent uneroded.</td>\n",
+       "      <td>the fame is untainted and talent uneroded.</td>\n",
+       "      <td>this day, the fame is untainted and the talent uneroded.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>35</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>the prince of wales described becoming a first-time grandfather as 'a unique moment in anyone's life', pictured at george's christening in 2013.</td>\n",
+       "      <td>prince of wales described becoming a first-time grandfather as 'a unique moment'</td>\n",
+       "      <td>prince of wales described as 'unique moment in anyone's life'</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>36</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>in order to preserve a pastime that requires vast open space in the midst of one of the world's densest regions, the club has managed to protect a patchwork of land that is larger than manhattan.</td>\n",
+       "      <td>the club has managed to preserve a pastime that requires vast open space.</td>\n",
+       "      <td>the club has managed to protect a pastime that is larger than manhattan.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>37</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>in stuart lancaster's fourth year in charge they are yet to win this competition.</td>\n",
+       "      <td>stuart lancaster's side are yet to win this competition.</td>\n",
+       "      <td>stuart lancaster's fourth year in charge they will win this competition.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>38</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>moores, a county championship winner with both sussex and lancashire, finds himself under increasing pressure to prove he has what it takes to succeed at international level after england's woeful showing at this world cup.</td>\n",
+       "      <td>moores, a county county championship winner, finds himself under increasing pressure.</td>\n",
+       "      <td>moores, a county championship, finds himself under increasing pressure.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>39</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>among the moulds he filmed using a specially designed set-up was part of the aspergillus genus, which commonly forms on starchy foods and are also used to help produce sake from rice.</td>\n",
+       "      <td>he filmed using a specially designed set-up part of the aspergillus genus.</td>\n",
+       "      <td>he filmed using a specially designed set-up was part of the aspergillus genus.</td>\n",
+       "      <td>B</td>\n",
+       "      <td>Both answers are equally probable.</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>40</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>the easter bunny was invented in german folklore, where an 'easter hare' was used to determine whether children had been well-behaved.</td>\n",
+       "      <td>the easter bunny was used to determine whether children had been well-behaved.</td>\n",
+       "      <td>the easter bunny was invented in german folklore, where children had been well-behaved.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>41</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>the scotland international made an # 11million move from leeds in the summer but the move has backfired with fulham falling perilously close to the relegation places in recent weeks.</td>\n",
+       "      <td>scotland international made an # 11million move from leeds in the summer.</td>\n",
+       "      <td>scotland international made an # 11million move from fulham in the summer.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>42</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>moores, a county championship winner with both sussex and lancashire, finds himself under increasing pressure to prove he has what it takes to succeed at international level after england's woeful showing at this world cup.</td>\n",
+       "      <td>moores, a county championship, finds himself under increasing pressure.</td>\n",
+       "      <td>moores, a county championship, finds himself under increasing pressure.</td>\n",
+       "      <td>B</td>\n",
+       "      <td>B</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>43</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>among the moulds he filmed using a specially designed set-up was part of the aspergillus genus, which commonly forms on starchy foods and are also used to help produce sake from rice.</td>\n",
+       "      <td>he filmed using a specially designed set-up part of the aspergillus genus.</td>\n",
+       "      <td>he filmed using a specially designed set-up was part of the genus genus.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>44</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>england coach mike catt wants england to be more ruthless when it comes to scoring points.</td>\n",
+       "      <td>mike catt wants england more ruthless when it comes to scoring points.</td>\n",
+       "      <td>mike catt wants england to be more ruthless when it comes.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>45</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>he was also ordered to pay $ 167 million in restitution and fines.</td>\n",
+       "      <td>he was also ordered to pay $ 167 million in restitution and fines.</td>\n",
+       "      <td>he was also ordered to pay $ 167 million in damages and fines.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>46</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>despite missing chances against scotland, for the first time in a while our strike runners consistently found clean air to run in.</td>\n",
+       "      <td>despite missing chances, runners consistently found clean air to run in.</td>\n",
+       "      <td>despite missing chances against scotland, for first time in a row.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>47</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>the prime minister's office had said tuesday that lee, 91, who is on a ventilator in the intensive care unit of singapore general hospital, had an infection and was being treated with antibiotics.</td>\n",
+       "      <td>lee, 91, is on a ventilator in the intensive care unit.</td>\n",
+       "      <td>lee, 91, is on a ventilator in the intensive care unit of singapore.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>A</td>\n",
+       "      <td>False</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>48</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>aubrey said she was also told her son would be eligible for a wheelchair when he is five years old.</td>\n",
+       "      <td>aubrey said he would be eligible when he is five years old.</td>\n",
+       "      <td>aubrey said she would be eligible when he is five years old.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>49</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>they made the material by etching features smaller than a wavelength of light onto a silicon film one thousand times thinner than a human hair.</td>\n",
+       "      <td>they made the material by etching features smaller than a wavelength of light.</td>\n",
+       "      <td>they made the material by etching features smaller than a wavelength of a silicon film.</td>\n",
+       "      <td>A</td>\n",
+       "      <td>B</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "      category   test_type  \\\n",
+       "0   factuality  order_bias   \n",
+       "1   factuality  order_bias   \n",
+       "2   factuality  order_bias   \n",
+       "3   factuality  order_bias   \n",
+       "4   factuality  order_bias   \n",
+       "5   factuality  order_bias   \n",
+       "6   factuality  order_bias   \n",
+       "7   factuality  order_bias   \n",
+       "8   factuality  order_bias   \n",
+       "9   factuality  order_bias   \n",
+       "10  factuality  order_bias   \n",
+       "11  factuality  order_bias   \n",
+       "12  factuality  order_bias   \n",
+       "13  factuality  order_bias   \n",
+       "14  factuality  order_bias   \n",
+       "15  factuality  order_bias   \n",
+       "16  factuality  order_bias   \n",
+       "17  factuality  order_bias   \n",
+       "18  factuality  order_bias   \n",
+       "19  factuality  order_bias   \n",
+       "20  factuality  order_bias   \n",
+       "21  factuality  order_bias   \n",
+       "22  factuality  order_bias   \n",
+       "23  factuality  order_bias   \n",
+       "24  factuality  order_bias   \n",
+       "25  factuality  order_bias   \n",
+       "26  factuality  order_bias   \n",
+       "27  factuality  order_bias   \n",
+       "28  factuality  order_bias   \n",
+       "29  factuality  order_bias   \n",
+       "30  factuality  order_bias   \n",
+       "31  factuality  order_bias   \n",
+       "32  factuality  order_bias   \n",
+       "33  factuality  order_bias   \n",
+       "34  factuality  order_bias   \n",
+       "35  factuality  order_bias   \n",
+       "36  factuality  order_bias   \n",
+       "37  factuality  order_bias   \n",
+       "38  factuality  order_bias   \n",
+       "39  factuality  order_bias   \n",
+       "40  factuality  order_bias   \n",
+       "41  factuality  order_bias   \n",
+       "42  factuality  order_bias   \n",
+       "43  factuality  order_bias   \n",
+       "44  factuality  order_bias   \n",
+       "45  factuality  order_bias   \n",
+       "46  factuality  order_bias   \n",
+       "47  factuality  order_bias   \n",
+       "48  factuality  order_bias   \n",
+       "49  factuality  order_bias   \n",
+       "\n",
+       "                                                                                                                                                                                                                          article_sentence  \\\n",
+       "0                                                                                                                        the abc have reported that those who receive centrelink payments made up half of radio rental's income last year.   \n",
+       "1                          five ambitious clubs are locked in a scramble for two champions league places behind chelsea and manchester city and it could prove more tense and exciting than the barclays premier league title race itself.   \n",
+       "2                                                                                 but it wasn't until last year that the 25 year old from pennsylvania went on to become a global superstar on a level with the likes of beyonce or adele.   \n",
+       "3                                                                                                                                                   seven games involving nimes were investigated after conrad was arrested last november.   \n",
+       "4                                                                                                                                                              the driver's side of the windscreen immediately shatters and falls through.   \n",
+       "5                   shafqat hussain, 24, was scheduled to be hanged on thursday, but a government official confirmed wednesday night with a tweet that hussain will not be hanged and that his case will be reviewed for another 72 hours.   \n",
+       "6                  ' referring to the ump's electoral alliance with the centrist union of democrats and independents (udi), mr sarkozy said: 'never in the history of the fifth republic has our political family won so many departments.   \n",
+       "7                                                                                                     insiders say the row brought simmering tensions between the starkly contrasting pair -- both rivals for miliband's ear -- to a head.   \n",
+       "8                 katarzyne pietrzycka, 42, was attacked with a stiletto by a thai transvestite jealous over her husband, leaving her bleeding heavily from her head (right) following the assault in the east coast thai city of pattaya.   \n",
+       "9                                             the man, named only as saleh, said the masked militant who appears in several beheading videos was a senior figure in the extremist organisation responsible for murdering foreign captives.   \n",
+       "10                                                                             the bolton boxer had hoped to take on floyd mayweather but the pound-for-pound king will instead meet manny pacquaio in a $ 300million mega-fight on may 2.   \n",
+       "11                                                                     the home which was built for former australian prime minister malcolm fraser and his wife tamie has been opened for inspection just a day after his sudden passing.   \n",
+       "12                 undeterred by the fact that he was unable to use his hands and lying next to several police officers, the man struggles to his feet while they are tackling two people nearby and sprints off through a watching crowd.   \n",
+       "13                           (cnn) the man who attacked a security area at the new orleans airport with a machete and wasp spray also had a bag of molotov cocktails and a car containing smoke bombs and gas cylinders, authorities said.   \n",
+       "14                                                                        former manchester city boss roberto mancini has turned up the heat on successor manuel pellegrini by claiming the chilean should be winning a trophy every year.   \n",
+       "15                                                                  but critics say they do not go far enough and will allow lenders to shut hundreds of branches across the country -- even when there is no other bank in the community.   \n",
+       "16                                                                                                                                                                       and a prized silver dollar each could fetch $ 10 million or more.   \n",
+       "17                   the toffees were the country's last hope on the continent before a 5-2 loss to dynamo kiev in europe on thursday night meant they followed chelsea, arsenal and manchester city in crashing out at the last 16 stage.   \n",
+       "18                                                                                                                                                 louis van gaal said that he believed wenger's estimation of the necessary points total.   \n",
+       "19                                                                                                                                      visitors to the isle of lewis will be hoping the clouds stay away on friday for the solar eclipse.   \n",
+       "20                                                                                                 despite their attempts to drown out the sounds, they claim customers are being driven away by the noises, which start around lunchtime.   \n",
+       "21                                                                                                                 captured at 1,500 frames per second, the video shows the chameleon attacking a cricket with remarkable speed and skill.   \n",
+       "22                                                                          mrs jarvis, who lives in the nearby village of farnham royal, said most of freya's friends have been allocated places at a school in the town of beaconsfield.   \n",
+       "23                                                                                                      despite missing chances against scotland, for the first time in a while our strike runners consistently found clean air to run in.   \n",
+       "24                                                                                                                         but now conditions on the farms that supply kfc with 23million chickens a year can be shown for the first time.   \n",
+       "25  both on screen and on the red carpet, she always radiates with a natural beauty - but while cate has avoided going under the knife or being jabbed with needles, she still relies on some beauty secrets to keep her looking her best.   \n",
+       "26                                                                                                                                                                                           she and her husband were trying to walk away.   \n",
+       "27                                     his former teacher, 35-year-old brianne altice, has been charged with 14 felonies - including rape, sodomy and sex abuse - for allegedly having sexual relationships with three different students.   \n",
+       "28                                                                                                                                          fans displayed the messages on bloomfield road ahead of their 3-0 defeat by charlton athletic.   \n",
+       "29                                                     we reported that, just before we battened down the hatches, all the three and four-year-old boys had been snapping at their pyjama elastic and comparing the size of their willies.   \n",
+       "30                                                                                                                                              england coach mike catt wants england to be more ruthless when it comes to scoring points.   \n",
+       "31                                                                            gomis, playing his second match since collapsing at white hart lane, struck three minutes from time at villa park to prick the bubble blown by tim sherwood.   \n",
+       "32                                   just load all the luggage you like in the boot and head to the coast, drive aboard the ferry, then start your holiday in relaxing surroundings, with great food, shopping and entertainment on board.   \n",
+       "33                                         herrera also revealed that captain wayne rooney is the joker inside united's dressing room, but admitted that he has to keep the conversation away from boxing, for fears rooney might hit him.   \n",
+       "34                                                                                                                                                                             to this day, the fame is untainted and the talent uneroded.   \n",
+       "35                                                                                        the prince of wales described becoming a first-time grandfather as 'a unique moment in anyone's life', pictured at george's christening in 2013.   \n",
+       "36                                     in order to preserve a pastime that requires vast open space in the midst of one of the world's densest regions, the club has managed to protect a patchwork of land that is larger than manhattan.   \n",
+       "37                                                                                                                                                       in stuart lancaster's fourth year in charge they are yet to win this competition.   \n",
+       "38         moores, a county championship winner with both sussex and lancashire, finds himself under increasing pressure to prove he has what it takes to succeed at international level after england's woeful showing at this world cup.   \n",
+       "39                                                 among the moulds he filmed using a specially designed set-up was part of the aspergillus genus, which commonly forms on starchy foods and are also used to help produce sake from rice.   \n",
+       "40                                                                                                  the easter bunny was invented in german folklore, where an 'easter hare' was used to determine whether children had been well-behaved.   \n",
+       "41                                                  the scotland international made an # 11million move from leeds in the summer but the move has backfired with fulham falling perilously close to the relegation places in recent weeks.   \n",
+       "42         moores, a county championship winner with both sussex and lancashire, finds himself under increasing pressure to prove he has what it takes to succeed at international level after england's woeful showing at this world cup.   \n",
+       "43                                                 among the moulds he filmed using a specially designed set-up was part of the aspergillus genus, which commonly forms on starchy foods and are also used to help produce sake from rice.   \n",
+       "44                                                                                                                                              england coach mike catt wants england to be more ruthless when it comes to scoring points.   \n",
+       "45                                                                                                                                                                      he was also ordered to pay $ 167 million in restitution and fines.   \n",
+       "46                                                                                                      despite missing chances against scotland, for the first time in a while our strike runners consistently found clean air to run in.   \n",
+       "47                                    the prime minister's office had said tuesday that lee, 91, who is on a ventilator in the intensive care unit of singapore general hospital, had an infection and was being treated with antibiotics.   \n",
+       "48                                                                                                                                     aubrey said she was also told her son would be eligible for a wheelchair when he is five years old.   \n",
+       "49                                                                                         they made the material by etching features smaller than a wavelength of light onto a silicon film one thousand times thinner than a human hair.   \n",
+       "\n",
+       "                                                                                                        correct_sentence  \\\n",
+       "0                                 those who receive centrelink payments made up half of radio rental's income last year.   \n",
+       "1                                         five ambitious clubs are locked in a scramble for two champions league places.   \n",
+       "2                                  the 25 year old from pennsylvania went on a level with the likes of beyonce or adele.   \n",
+       "3                                               seven games involving nimes were investigated after conrad was arrested.   \n",
+       "4                                                        driver's side of the windscreen immediately shatters and falls.   \n",
+       "5                                                              shafqat hussain, 24, was scheduled to be hanged thursday.   \n",
+       "6                                                       sarkozy's electoral alliance has won the departmental elections.   \n",
+       "7                                                    insiders say the row brought tensions between the contrasting pair.   \n",
+       "8                                   katarzyne pietrzycka, 42, was attacked with a stiletto by thai transvestite jealous.   \n",
+       "9                            the man, named only as saleh, said the masked militant appears in several beheading videos.   \n",
+       "10                                              the boxer will instead meet manny pacquaio in a $ 300million mega-fight.   \n",
+       "11                                               the home was built for former australian prime minister malcolm fraser.   \n",
+       "12                                             he was unable to use his hands and lying next to several police officers.   \n",
+       "13  the man who attacked a security area at the new orleans airport was with a machete and a car containing smoke bombs.   \n",
+       "14                                              roberto mancini has turned up the heat on manuel pellegrini's successor.   \n",
+       "15                                        critics say they do not go far enough and will allow lenders to shut hundreds.   \n",
+       "16                                                                 a prized silver dollar each could fetch $ 10 million.   \n",
+       "17                                                           the toffees were the country's last hope on thursday night.   \n",
+       "18                                          louis van gaal says he believed wenger's estimation of the necessary points.   \n",
+       "19                                          visitors to the isle of lewis will be hoping the clouds stay away on friday.   \n",
+       "20                                                             they claim customers are being driven away by the noises.   \n",
+       "21                                                                                  captured at 1,500 frames per second.   \n",
+       "22                                      she said freya's friends have been allocated places at a school in beaconsfield.   \n",
+       "23                                              despite missing chances, runners consistently found clean air to run in.   \n",
+       "24                                                      but now conditions on the farms can be shown for the first time.   \n",
+       "25                                              she always radiates with a natural beauty, but cate is looking her best.   \n",
+       "26                                                                             she and husband were trying to walk away.   \n",
+       "27                                brianne altice, 35-year-old brianne, has been charged with rape, sodomy and sex abuse.   \n",
+       "28                                             fans displayed the messages on bloomfield road ahead of their 3-0 defeat.   \n",
+       "29                                               the three and four-year-old boys were snapping at their pyjama elastic.   \n",
+       "30                                                                    coach mike catt wants england to be more ruthless.   \n",
+       "31                                                                   gomis struck three minutes from time at villa park.   \n",
+       "32                                       the luggage you like in the boot and head to the coast, drive aboard the ferry.   \n",
+       "33                                         herrera also revealed that rooney is the joker inside united's dressing room.   \n",
+       "34                                                                            the fame is untainted and talent uneroded.   \n",
+       "35                                      prince of wales described becoming a first-time grandfather as 'a unique moment'   \n",
+       "36                                             the club has managed to preserve a pastime that requires vast open space.   \n",
+       "37                                                              stuart lancaster's side are yet to win this competition.   \n",
+       "38                                 moores, a county county championship winner, finds himself under increasing pressure.   \n",
+       "39                                            he filmed using a specially designed set-up part of the aspergillus genus.   \n",
+       "40                                        the easter bunny was used to determine whether children had been well-behaved.   \n",
+       "41                                             scotland international made an # 11million move from leeds in the summer.   \n",
+       "42                                               moores, a county championship, finds himself under increasing pressure.   \n",
+       "43                                            he filmed using a specially designed set-up part of the aspergillus genus.   \n",
+       "44                                                mike catt wants england more ruthless when it comes to scoring points.   \n",
+       "45                                                    he was also ordered to pay $ 167 million in restitution and fines.   \n",
+       "46                                              despite missing chances, runners consistently found clean air to run in.   \n",
+       "47                                                               lee, 91, is on a ventilator in the intensive care unit.   \n",
+       "48                                                           aubrey said he would be eligible when he is five years old.   \n",
+       "49                                        they made the material by etching features smaller than a wavelength of light.   \n",
+       "\n",
+       "                                                                                                  incorrect_sentence  \\\n",
+       "0          the abc have reported that those who receive centrelink payments made up radio rental's income last year.   \n",
+       "1                                          five ambitious clubs are locked in a bid for two champions league places.   \n",
+       "2                        the 25 year old from pennsylvania went viral on a level with the likes of beyonce or adele.   \n",
+       "3                                                           seven games involving nimes were arrested last november.   \n",
+       "4                                                         driver's side were immediately shatters and falls through.   \n",
+       "5                                                      shafqat hussain, 24, was scheduled to be hanged on wednesday.   \n",
+       "6                                                       sarkozy's electoral alliance has won departmental elections.   \n",
+       "7                                            insiders say the row brought simmering tensions between miliband's ear.   \n",
+       "8                                            katarzyne pietrzycka, 42, was attacked with a stiletto by a thai court.   \n",
+       "9                                                 the man, named only as saleh, appears in several beheading videos.   \n",
+       "10                             the bolton boxer had hoped to take the floyd mayweather in a $ 300million mega-fight.   \n",
+       "11                                            the home was built for inspection just a day after his sudden passing.   \n",
+       "12                                             the man struggles to use his hands and lying next to police officers.   \n",
+       "13  the man who attacked a security area at the new orleans airport with a machete and a car containing smoke bombs.   \n",
+       "14                                         roberto mancini has turned up the heat for manuel pellegrini's successor.   \n",
+       "15                                               critics say they do not go far enough to shut hundreds of branches.   \n",
+       "16                                                             a prized silver dollar each could fetch $ 10 or more.   \n",
+       "17                                                            the toffees were last 16 in a 5-2 loss to dynamo kiev.   \n",
+       "18                                            louis van gaal says wenger's estimation of the necessary points total.   \n",
+       "19                                              visitors to the isle of lewis will be hoping to stay away on friday.   \n",
+       "20                                                              they claim they are being driven away by the noises.   \n",
+       "21                                                                              captured by 1,500 frames per second.   \n",
+       "22                                       mrs jarvis said most of her friends have been allocated places at a school.   \n",
+       "23                                               despite missing chances against scotland, for first time in a year.   \n",
+       "24                                               kfc with 23million chickens a year can be shown for the first time.   \n",
+       "25                                     she always radiates with needles, but cate has avoided going under the knife.   \n",
+       "26                                                                        she and her wife were trying to walk away.   \n",
+       "27                                             brianne altice, 21, has been charged with rape, sodomy and sex abuse.   \n",
+       "28                                            fans displayed the messages on bloomfield road ahead of their 3-0 win.   \n",
+       "29                                                     the three and boys had been snapping at their pyjama elastic.   \n",
+       "30                                                        mike catt wants england to be more ruthless when it comes.   \n",
+       "31                                                          gomis scored three goals since collapsing at villa park.   \n",
+       "32                                             the ferry, then start your holiday in the boot and head to the coast.   \n",
+       "33                                  herrera also revealed that rooney has to keep the conversation away from boxing.   \n",
+       "34                                                          this day, the fame is untainted and the talent uneroded.   \n",
+       "35                                                     prince of wales described as 'unique moment in anyone's life'   \n",
+       "36                                          the club has managed to protect a pastime that is larger than manhattan.   \n",
+       "37                                          stuart lancaster's fourth year in charge they will win this competition.   \n",
+       "38                                           moores, a county championship, finds himself under increasing pressure.   \n",
+       "39                                    he filmed using a specially designed set-up was part of the aspergillus genus.   \n",
+       "40                           the easter bunny was invented in german folklore, where children had been well-behaved.   \n",
+       "41                                        scotland international made an # 11million move from fulham in the summer.   \n",
+       "42                                           moores, a county championship, finds himself under increasing pressure.   \n",
+       "43                                          he filmed using a specially designed set-up was part of the genus genus.   \n",
+       "44                                                        mike catt wants england to be more ruthless when it comes.   \n",
+       "45                                                    he was also ordered to pay $ 167 million in damages and fines.   \n",
+       "46                                                despite missing chances against scotland, for first time in a row.   \n",
+       "47                                              lee, 91, is on a ventilator in the intensive care unit of singapore.   \n",
+       "48                                                      aubrey said she would be eligible when he is five years old.   \n",
+       "49                           they made the material by etching features smaller than a wavelength of a silicon film.   \n",
+       "\n",
+       "   result                      swapped_result   pass  \n",
+       "0       A                                   B   True  \n",
+       "1       A                                   B   True  \n",
+       "2       A                                   B   True  \n",
+       "3       A                                   B   True  \n",
+       "4       A                                   A  False  \n",
+       "5       A                                   B   True  \n",
+       "6       A                                   A  False  \n",
+       "7       A                                   B   True  \n",
+       "8       A                                   B   True  \n",
+       "9       A                                   B   True  \n",
+       "10      A                                   B   True  \n",
+       "11      A                                   B   True  \n",
+       "12      B                                   B  False  \n",
+       "13      B                                   B  False  \n",
+       "14      A                                   B   True  \n",
+       "15      B                                   B  False  \n",
+       "16      A                                   B   True  \n",
+       "17      B                                   A  False  \n",
+       "18      A                                   A  False  \n",
+       "19      A                                   B   True  \n",
+       "20      A                                   B   True  \n",
+       "21      A                                   B   True  \n",
+       "22      A                                   B   True  \n",
+       "23      A                                   B   True  \n",
+       "24      A                                   B   True  \n",
+       "25      A                                   B   True  \n",
+       "26      A                                   B   True  \n",
+       "27      A                                   B   True  \n",
+       "28      A                                   B   True  \n",
+       "29      A                                   B   True  \n",
+       "30      A                                   B   True  \n",
+       "31      A                                   B   True  \n",
+       "32      B                                   B  False  \n",
+       "33      A                                   B   True  \n",
+       "34      A                                   B   True  \n",
+       "35      A                                   B   True  \n",
+       "36      A                                   B   True  \n",
+       "37      A                                   B   True  \n",
+       "38      A                                   B   True  \n",
+       "39      B  Both answers are equally probable.  False  \n",
+       "40      A                                   B   True  \n",
+       "41      A                                   B   True  \n",
+       "42      B                                   B  False  \n",
+       "43      A                                   B   True  \n",
+       "44      A                                   B   True  \n",
+       "45      A                                   B   True  \n",
+       "46      A                                   B   True  \n",
+       "47      A                                   A  False  \n",
+       "48      A                                   B   True  \n",
+       "49      A                                   B   True  "
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This method returns the generated results in the form of a pandas dataframe, which provides a convenient and easy-to-use format for working with the test results. You can use this method to quickly identify the test cases that failed and to determine where fixes are needed."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Final Results\n",
+    "We can call `.report()` which summarizes the results giving information about pass and fail counts and overall test pass/fail flag."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>category</th>\n",
+       "      <th>test_type</th>\n",
+       "      <th>fail_count</th>\n",
+       "      <th>pass_count</th>\n",
+       "      <th>pass_rate</th>\n",
+       "      <th>minimum_pass_rate</th>\n",
+       "      <th>pass</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>factuality</td>\n",
+       "      <td>order_bias</td>\n",
+       "      <td>11</td>\n",
+       "      <td>39</td>\n",
+       "      <td>78%</td>\n",
+       "      <td>70%</td>\n",
+       "      <td>True</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "     category   test_type  fail_count  pass_count pass_rate minimum_pass_rate  \\\n",
+       "0  factuality  order_bias          11          39       78%               70%   \n",
+       "\n",
+       "   pass  \n",
+       "0  True  "
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "harness.report()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Bias Analysis and Accuracy Visualization\n",
+    "\n",
+    "In this section, we conduct an analysis to identify biases and assess accuracy.\n",
+    "#### Bias Identification\n",
+    "\n",
+    "We identify bias in the responses based on specific patterns:\n",
+    "\n",
+    "- **Bias Towards A**: Occurs when both the \"result\" and \"swapped_result\" are \"A.\" This bias is in favor of \"A,\" but it's incorrect, so it's marked as **False**.\n",
+    "- **Bias Towards B**: Occurs when both the \"result\" and \"swapped_result\" are \"B.\" This bias is in favor of \"B,\" but it's incorrect, so it's marked as **False**.\n",
+    "- **No Bias** : When \"result\" is \"B\" and \"swapped_result\" is \"A,\" there is no bias. However, this statement is incorrect, so it's marked as **False**.\n",
+    "- **No Bias** : When \"result\" is \"A\" and \"swapped_result\" is \"B,\" there is no bias. This statement is correct, so it's marked as **True**.\n",
+    "\n",
+    "#### Accuracy Assessment\n",
+    "\n",
+    "Accuracy is assessed by examining the \"pass\" column. If \"pass\" is marked as **True**, it indicates a correct response. Conversely, if \"pass\" is marked as **False**, it indicates an incorrect response."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import matplotlib.pyplot as plt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAEiCAYAAAAPh11JAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABNVUlEQVR4nO3deVyN6f8/8NfpqNOqtJIoQmKIUsyQaiwNNZYRoghjGft3LIMMLYYZY2xjrB9GZpStscxYJ3szGvsug5BljBaKlLRcvz88un+Oc8qdQdLr+Xj04Fz3dd/nfZ/uU6/u676voxBCCBARERHRC+mUdQFERERE5QWDExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxNVGN7e3lAoFGVdRpmLioqCQqFAVFRUWZdSZvgaENHLYnCicuf69etQKBQaX0ZGRmjcuDEiIiKQlZVV1mUW6/m6K1WqBBsbG/j7+2P37t3/eftFr0+/fv3+e7Gk4eDBg9L3bsOGDWVdDhG9YZXKugCil+Xo6Ijg4GAAgBACqamp2LFjB8LDw7Fz50788ccfUCqVUv+ffvoJ2dnZZVWuGgsLC4wYMQIA8PjxY5w/fx7btm3Dtm3bEBMTg169er225+7atStatGiBatWqvbbneJetWLECwNMA/OOPP6J79+5lXBERvUkMTlRu1alTB+Hh4Wptubm5eP/99/HXX3/hwIED+PDDD6VlNWvWfMMVFs/S0lKj9rVr16JXr16YNGnSaw1OpqamMDU1fW3bf5c9ePAAsbGxaNy4MWxsbPD777/j5s2bqFGjRlmXRkRvCIfq6J2iUqng4+MDAEhLS1Nbpu0ap8zMTMycORNeXl6wtbWFnp4ebG1t0bdvXyQlJWls//Hjx5g9ezZcXFxgamoKIyMjODg4oEePHjh9+vR/qr1nz54wMjJCcnKyRu2bNm1Cr169UKdOHRgaGsLU1BSenp745Zdf1PpFRUWhVq1aAIBVq1apDQnu379f6lPc9T1//vkn/Pz8YG5uDn19fdSvXx9hYWFaz9SdOHECAQEBqFmzJlQqFaysrODu7o7p06fL2t9Lly7hiy++gKurKywsLKCvr4969eph4sSJWodai75/eXl5CA8Ph4ODA1QqFerVq4dFixZpfY579+7hs88+g42NDQwNDeHu7o5NmzbJqk+bNWvWIDs7G3379kXfvn1RWFhY4nVSKSkpGDt2LJycnGBgYABzc3M0b94c3333nUbf06dPIygoCHZ2dlCpVKhWrRo++ugj/Pbbb1Kf8PBwte/ls7R9X58dtk1MTETXrl1hYWEBhUKB69evA5B/bJWm1uXLl0OhUODbb7/Vuv7evXuhUCgwZMiQYp+D6G3FM070Tnny5An2798PhUKBJk2avLB/YmIipk6dCh8fH3Tt2hVGRka4ePEiYmJisG3bNpw4cQL29vZS/5CQEKxfvx6NGzdG//79oVKpcPPmTezbtw9Hjx6Fi4vLK9mPSpXU35qTJk2Cnp4eWrVqhWrVqiE1NRW//vorAgIC8P3332PkyJEAgCZNmmD06NGYP38+XFxc0KVLF2kbDg4OJT7nhg0b0KtXL6hUKvTs2RPW1tb4/fffERkZiV27dmH//v3Q19cHAJw6dQoffPABlEolOnfuDHt7e2RkZODChQtYtmwZJk+e/MJ93LhxI1asWAEfHx94e3ujsLAQf/31F2bOnIkDBw7g4MGD0NXV1VivV69eOHLkCDp06AClUon169dj+PDh0NXVxaBBg6R+2dnZ8Pb2xtmzZ/H+++/Dy8sLN2/eRM+ePdG+ffsX1qfNihUroFQqERQUhMqVK2Po0KFYuXIlvvzyS41Q/vfff8PHxwd37txBq1at0KVLFzx69Ajnz5/HjBkzMG7cOKnvL7/8gt69e0MIgY8//hhOTk5ISUnB4cOHsWLFCnz88ccvVW+RK1euoEWLFmjUqBH69euH9PR06OnpAZB/bJWm1l69emHs2LFYsWIFvvjiC416/ve//wGA2veLqNwQROXMtWvXBADh6OgowsLCRFhYmJg6daoYNmyYcHR0FPr6+mLWrFka63l5eYnnD/mMjAyRnp6u0Xfv3r1CR0dHDBw4UK2vQqEQbm5uIj8/X61/fn6+uH//vqz6AQgnJyeN9piYGAFANGzYUGNZUlKSRtvDhw9Fo0aNhKmpqXj06JHUXvT6hISEaH3+lStXCgBi5cqVUltmZqYwNTUVKpVKnD59WmovKCgQPXv2FABEZGSk1D5mzBgBQGzevFlj+2lpaVqf93m3bt0Subm5Gu0RERECgFi9erVae9H3r3nz5iIzM1Nqv3jxoqhUqZLGaxoWFiYAiEGDBqm179y5UwDQeA1e5MyZMwKA8PX1ldr69u0rAIjdu3dr9G/WrJkAIJYtW6ax7ObNm9L///33X2FkZCSMjIzEiRMnSuxbtE/79u3T6Kft+1p0LAAQU6dO1bpfpTm2SlPr0KFDBQCxf/9+tT7p6elCpVKJJk2aaK2H6G3H4ETlzrO/DLR9+fv7i5MnT2qspy04laRRo0bCwcFBepyZmSkAiJYtW4rCwsKXrh+AsLCwkELfhAkThL+/v1AoFMLY2FgcPHhQ9rZmz56t8cvpZYLTTz/9JACIoUOHavRPTk4WlSpVErVr15baioLTrl27ZNcqV3p6ugAg+vXrp9Ze9P3bu3evxjpFyx48eCC11apVS+jp6Yk7d+5o9G/Tpk2pg9Po0aMFABEdHS217d69WwAQvXr1Uut7+PBhAUC0bt36hdudOXNmicHmWS8bnKpWrao1pJZE27FVmlpPnz4tAIjg4GC19nnz5gkAYuHChaWqh+htwWucqNzy9fWFeBr+IYRAWloatmzZgnPnzqFly5Y4fPiwrO3s378fXbp0QbVq1aCrqytdE3T27Fn8888/Ur/KlSujY8eO+PPPP+Hq6ooZM2bg0KFDyMvLK3Xt6enpiIiIQEREBGbOnImtW7fCyMgIcXFx8PT01OifkpKCMWPGwNnZGYaGhlKNY8eOBQC1Ol/GyZMnATy9juh5NWvWRO3atXH16lU8fPgQANCjRw/o6Oiga9euGDBgANasWYPbt2+X6jmFEPjxxx/RunVrmJubQ6lUQqFQwMLCosR9cnNz02izs7MDAGRkZAB4ehH3tWvXUKdOHVStWlWjv7bXuCS5ublYvXo1TExM0LVrV6ndx8cHNWrUwKZNm3D//n2p/ciRIwAga0iwNH1flouLizQ097zSHFulqbVx48Zo0aIFYmNjpe8L8HS409DQEEFBQf9hj4jKDq9xoneGhYUFOnXqBENDQ7Rr1w5ffvkl4uLiSlxnw4YN6NmzJ4yNjeHr6wsHBwfpl0dUVBSSk5M1+s+YMQMxMTHSdTyVK1dG//79MWPGDBgaGsqq1cnJCRcvXgTw9Jf95s2bMXToUHTt2hXHjh1D9erVpb737t2Du7s7bty4gZYtW6Jt27YwMzODUqnEqVOnsGXLFuTm5pbmpdLw4MEDAICNjY3W5dWqVcOlS5fw4MEDmJiYoHnz5ti/f7/0WqxcuRIA4O7ujpkzZ0oX6Jdk1KhR+OGHH1CjRg106tQJ1apVg0qlAgBEREQUu0+VK1fWaCu6JqygoEBtf6ytrbVuo7j9LM7mzZuRnp6O/v37w8DAQGrX0dFBUFAQvvnmG8TExGD48OEAnt50AEDt+1ic0vR9WcXtb2mPrdLWOmTIEPTv3x+rV6/GiBEjcPjwYZw9exYhISG8s5PKLQYneuc0b94cAHD06NEX9g0PD4e+vj6OHz+OunXrqi1bu3atRn9DQ0N89dVX+Oqrr3Dt2jXs27cPS5Yswfz585GTk4OlS5eWul4zMzP069cPBQUFGDhwIIYPH47NmzdLy1esWIEbN25g2rRp+PLLL9XW/eabb7Bly5ZSP+fzisLI3bt3tS7/999/1foBT8/a7NixAzk5OTh8+DB+++03LFq0CH5+fjh37hxq165d7POlpKRg4cKFaNy4MRISEtQC57///ouIiIhXsj8pKSlalxe3n8Upmrtp5cqVUkjU1qcoOJmZmQGArLNwz/Z90QX8OjpPBwny8/M1lhWFGm2KmzG/tMdWaWoFnt4p+vnnn2P58uUYMWIEli9fDoAXhVP5xqE6eucUDZkUFha+sG9SUhKcnZ01QtOdO3dw9erVEtetVasWBgwYgAMHDsDY2Bi//vrryxcNYMCAAXB1dcWWLVtw6NAhtRoBoHPnzhrrxMfHa7QVTfpZdPZFjqZNmwKA1tvcb968iaSkJNSuXRsmJiYayw0MDODt7Y3Zs2cjNDQUOTk5LzzTd/XqVQgh0LZtW42zdNr2qbQqV66MWrVq4cqVK1Loe9nnSE5Oxp49e2BjY4NPP/1U61etWrVw8uRJacjTw8MDAPD777+/cPul6VulShUA2gNZ0XOXRmmPrdLUCjw9Nvr27YvTp09j3759WLduHZydndGyZctS10r0tmBwonfOnDlzAACtW7d+YV97e3tcuXJF7QzE48ePMXToUI1rl1JTU3Hu3DmNbdy/fx+5ubnSrfovS6FQICwsDAAwZcoUtRoB4I8//lDrHxMTg+3bt2tsp0qVKlAoFLh586bs5+7cuTNMTU2xcuVKnD9/XmoXQmDChAnIz89X+wiXhIQEPH78WGM7Ra/ji16Lon06dOiQWsC9desWJk2aJLvukvTp0wdPnjzB1KlT1dp///137NmzR/Z2Vq5cicLCQgwZMgTLly/X+jVx4kQA///MlLu7O9zd3XHw4EHp1vtnPRt8QkJCYGxsjNmzZ+PUqVMl9nV3dwfwdBb8Z1+3hIQEREdHy96nIqU9tkpTa5GiuZqCg4Px8OFDnm2i8q8sr0wnehnapiMICwsTI0eOFK6urgKAqFKlijh37pzaetruqluwYIEAIKpVqyZGjhwphg4dKurUqSMcHR2Fi4uLWv+TJ08KAMLFxUX06dNHTJw4UQwaNEhYW1sLAOKHH36QVT+KmY6giJubm9rdTDdv3hSmpqZCqVSK7t27i3Hjxol27doJHR0d8cknn2i9O8zDw0MoFAoRHBwsIiIixLRp08T169eFENrvvhJCiPXr1wulUimMjIzEgAEDxIQJE6RaPDw8RE5OjtS3c+fOonLlysLf31+MHDlSjB8/XrpTrXbt2mrTBRSnW7duAoBo2rSpGDdunOjTp4+oUqWKCAgIEACEl5eXWv+S7ooMCQkRAMS1a9ektqysLPHee+8JAOKDDz4QEydOFEFBQUJXV1f4+fnJuquuoKBA1KxZUygUCnH16tVi+2VkZAgDAwNhZmYmvU6XLl0Stra2AoDw9PQUX3zxhRg1apRo06aNMDc3V1t/48aNQk9PT+jq6opu3bqJ0NBQMXjwYOHi4iI6d+6s1rdly5bS92TcuHGie/fuQk9PT3Tt2rXYu+qKu8PyZY6t0tRaxNPTUwAQKpVK9nQVRG8rBicqd4qbjkClUglHR0cxdOhQkZycrLGetl+8hYWFYsmSJaJhw4ZCX19fVK1aVXz66aciJSVFo//9+/dFeHi4aN26tahWrZrQ09MTtra24qOPPhI7duyQXf+LgtNvv/0m/bItcurUKdG+fXtRpUoVYWJiIry8vMTu3buLDUF///236NixozAzMxMKhULtFvbi1hFCiIMHD4oOHToIMzMzoaenJ+rVqyemTJkisrKy1Prt3LlT9O3bVzg5OQkTExNhbGwsGjRoIEJDQ0Vqaqqs1+Hhw4di7NixwsHBQahUKlG3bl0xbdo08eTJk1cSnIR4OrXB4MGDhZWVldDX1xdubm5i48aNJb4Gz9q1a5fWWrQJCgrSmK7g33//FaNHjxa1a9cWenp6wtzcXDRv3lzMmTNHY/2TJ0+KHj16CBsbG6GrqyuqVasmOnToILZu3arWLy0tTfTt21eYm5sLAwMD0aJFC7Fr164SpyMoLjgJUfpjqzS1Flm+fLkAIAIDA0t+EYnKAYUQQryGE1lEREQAgBEjRmDhwoXYs2eP2udHEpVHDE5ERPTapKamonbt2qhevToSExOLvcOPqLzgdARERPTKFX3WY2xsLLKysqQPKCYq7xiciIjolduwYQNWrVoFW1tbzJgxA4GBgWVdEtErwaE6IiIiIpk4jxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCRTpbIugN6sGzduIC0trazLIHqh3NxcqFSqsi6D6IUsLS1Rs2bNsi6D3hAGpwrkxo0bcHZ2RnZ2dlmXQvRCSqUSBQUFZV0G0QsZGhoiMTGR4amCYHCqQNLS0pCdnY3Vq1fD2dm5rMshKtb27dsxZcoUHqv01ktMTERwcDDS0tIYnCoIBqcKyNnZGa6urmVdBlGxEhMTAfBYJaK3Dy8OJyIiIpKJwYmIiIhIJgYnojIQFRUFhUKB69evl3UpVAFcv34dCoUCUVFRZV0KUbnH4ETvvKKQolAo8Mcff2gsF0KgRo0aUCgU8Pf3L/X2Fy1axF9I9Ep16tQJhoaGePjwYbF9goKCoKenh/T09DdYGRExOFGFoa+vj5iYGI32AwcO4NatWy89Z9DLBKc+ffogJycH9vb2L/Wc9G4LCgpCTk4ONm3apHV5dnY2tmzZgo8++ggWFhZvuDqiio3BiSqMjh07YsOGDcjPz1drj4mJgZubG6pWrfraa3j06BGAp3MU6evrQ6FQvPbnpPKnU6dOMDEx0Rr0AWDLli149OgRgoKC3nBlRMTgRBVGr169kJ6ejri4OKntyZMniI2NRe/evTX6FxYWYt68eWjYsCH09fVhY2ODIUOG4P79+1IfBwcHnD9/HgcOHJCGA729vQH8/yHCAwcOYNiwYbC2toadnZ3asuevcdqxYwe8vLxgYmKCypUrw93dvdhfnvTuMjAwwCeffII9e/YgJSVFY3lMTAxMTEzQqlUrjBs3Do0aNYKxsTEqV66MDh064PTp0y98Dm9vb+lYfVa/fv3g4OCg1ibnvUBUUTA4UYXh4OCA999/H2vWrJHaduzYgczMTAQGBmr0HzJkCMaPH4+WLVti/vz56N+/P6Kjo+Hr64u8vDwAwLx582BnZ4f69evj559/xs8//4zJkyerbWfYsGG4cOECpk6diokTJxZbX1RUFPz8/HDv3j1MmjQJ33zzDZo0aYKdO3e+oleAypOgoCDk5+dj/fr1au337t3Drl270LVrV9y5cwebN2+Gv78/5syZg/Hjx+Ps2bPw8vLCP//888pqkfNeIKooOAEmVSi9e/fGpEmTkJOTAwMDA0RHR8PLywu2trZq/f744w8sX74c0dHRamejfHx88NFHH2HDhg3o3bs3unTpgi+//BKWlpYIDg7W+pzm5ubYs2cPlEplsXVlZmZi1KhR8PDwwP79+6Gvry8tE0L8x72m8ujDDz9EtWrVEBMTgxEjRkjtGzZsQF5eHoKCgtCoUSNcunQJOjr//2/gPn36oH79+lixYgWmTJnyn+uQ+14gqih4xokqlB49eiAnJwdbt27Fw4cPsXXrVq0/9Dds2ABTU1O0a9cOaWlp0pebmxuMjY2xb98+2c85aNCgEkMTAMTFxeHhw4eYOHGiWmgCwOugKiilUonAwEAkJCSoDenGxMTAxsYGbdq0gUqlkkJTQUEB0tPTYWxsDCcnJ5w4ceKV1PEq3wtE7wIGJ6pQrKys0LZtW8TExGDjxo0oKChAQECARr/Lly8jMzMT1tbWsLKyUvvKysrSet1JcWrVqvXCPklJSQCA9957T/7O0Duv6OLvouvcbt26hfj4eAQGBkKpVKKwsBBz585F3bp1oVKpYGlpCSsrK5w5cwaZmZmvpIZX+V4gehdwqI4qnN69e2PQoEH4999/0aFDB5iZmWn0KSwshLW1NaKjo7Vuw8rKSvbzGRgYvGypVMG5ubmhfv36WLNmDUJDQ7FmzRoIIaRANWPGDEyZMgUDBgzAtGnTYG5uDh0dHfzf//0fCgsLS9y2QqHQOgxcUFCg9vhVvheI3gUMTlThdO3aFUOGDMFff/2FdevWae3j6OiI3bt3o2XLli8MPq9iKM3R0REAcO7cOdSpU+c/b4/eHUFBQZgyZQrOnDmDmJgY1K1bF+7u7gCA2NhY+Pj4YMWKFWrrZGRkwNLSssTtVqlSBVevXtVoT05OVntcmvcCUUXAoTqqcIyNjbF48WKEh4fj448/1tqnR48eKCgowLRp0zSW5efnIyMjQ3psZGSk9vhltG/fHiYmJvj666/x+PFjtWW8OLxiKzq7NHXqVJw6dUpt7ialUqlxfGzYsAG3b99+4XYdHR1x8eJFpKamSm2nT5/Gn3/+qdavNO8FooqAZ5yoQgoJCSlxuZeXF4YMGYKvv/4ap06dQvv27aGrq4vLly9jw4YNmD9/vnRtlJubGxYvXoyvvvoKderUgbW1NT788MNS1VO5cmXMnTsXAwcOhLu7O3r37o0qVarg9OnTyM7OxqpVq156X6l8q1WrFj744ANs2bIFANSCk7+/PyIjI9G/f3988MEHOHv2LKKjo1G7du0XbnfAgAGYM2cOfH198emnnyIlJQVLlixBw4YN8eDBA6lfad4LRBUBgxNRMZYsWQI3NzcsXboUoaGhqFSpEhwcHBAcHIyWLVtK/aZOnYrk5GR8++23ePjwIby8vEodnADg008/hbW1Nb755htMmzYNurq6qF+/Pj7//PNXuVtUDgUFBeHQoUPw8PBQG8oNDQ3Fo0ePEBMTg3Xr1sHV1RXbtm0rcb6wIs7Ozvjpp58wdepUjBkzBg0aNMDPP/+MmJgY7N+/X62v3PcCUUWgEBwHqDBOnDgBNzc3HD9+HK6urmVdDlGxoqOjERwczGOV3nr8uVrx8BonIiIiIpkYnIiIiIhkYnAiIiIikonBiYiIiEgmBiciIiIimRiciIiIiGTiPE4V0Pbt25GYmFjWZRAVq2j2ah6r9La7du1aWZdAbxjncapAEhIS4OnpqfEhnkRvIx0dnRd+UC3R20CpVCI+Ph7vv/9+WZdCbwDPOFUgKpUKBQUFWL16NZydncu6HKJibd++HVOmTOGxSm+9xMREBAcHQ6VSlXUp9IYwOFVAzs7OnOGW3mpFw3M8VonobcOLw4mIiIhkYnCiMqdQKBAeHl7WZZR74eHhUCgUZV0GEdE7jcGJXrmoqCgoFAq1L2tra/j4+GDHjh1lWpuDg4NGbdq+oqKiyrTOt0VBQQFsbW2hUCjK/HtXEVy+fBmBgYGws7ODoaEh6tevj8jISGRnZ79w3du3b6NHjx4wMzND5cqV0blzZ1y9elVr3xUrVsDZ2Rn6+vqoW7cuFixYoNHnzz//hKurK0xMTODt7Y2LFy9q9Bk1ahR8fX1Lv6NE5RivcaLXJjIyErVq1YIQAnfv3kVUVBQ6duyI3377Df7+/lK/nJwcVKr0Zg7FefPmISsrS3q8fft2rFmzBnPnzoWlpaXU/sEHH7yRet52e/fuxZ07d+Dg4IDo6Gh06NChrEt6Z928eRMeHh4wNTXFiBEjYG5ujoSEBISFheH48ePYsmVLsetmZWXBx8cHmZmZCA0Nha6uLubOnQsvLy+cOnUKFhYWUt+lS5fis88+Q7du3TBmzBjEx8dj1KhRyM7OxoQJEwAAmZmZ6Ny5M1q0aIHBgwcjKioK3bp1w5kzZ6BUKgEA58+fx//+9z8cP3789b4wRG8bQRXG8ePHBQBx/Pjx1/o8K1euFADE0aNH1drv3bsndHV1Re/evV/r85fGrFmzBABx7dq1si7lhfLy8kRubm6xy8PCwsSrfkv37dtXuLq6ivnz5wsjIyORlZX1SrdfnNWrV7+RY/VtMn36dAFAnDt3Tq29b9++AoC4d+9esevOnDlTABBHjhyR2hITE4VSqRSTJk2S2rKzs4WFhYXw8/NTWz8oKEgYGRlJz7Fjxw5haGgocnJyhBBCXLt2TQAQFy9elNZp27atGDly5Mvv8DviTf1cpbcHh+rojTEzM4OBgYHG2aXnr3FKTk7GsGHD4OTkBAMDA1hYWKB79+64fv262np5eXmIiIhA3bp1oa+vDwsLC7Rq1QpxcXH/qc78/HxMmzYNjo6OUKlUcHBwQGhoKHJzc6U+Y8aMgYWFBcQz06CNHDkSCoUC33//vdR29+5dKBQKLF68GADw5MkTTJ06FW5ubjA1NYWRkRE8PT2xb98+tRquX78OhUKB7777DvPmzZNquXDhAgDgjz/+gLu7O/T19eHo6IilS5dq3Ze4uDi0atUKZmZmMDY2hpOTE0JDQ2W9Djk5Odi0aRMCAwPRo0cP5OTklHjWg/6bBw8eAABsbGzU2qtVqwYdHR3o6ekVu25sbCzc3d3h7u4utdWvXx9t2rTB+vXrpbZ9+/YhPT0dw4YNU1t/+PDhePToEbZt2wbg6fdeX18f+vr6AABzc3MAkIYMN2/ejJMnTyIiIuJld5eo3GJwotcmMzMTaWlpSE1Nxfnz5zF06FBkZWUhODi4xPWOHj2KQ4cOITAwEN9//z0+++wz7NmzB97e3mrXeoSHhyMiIgI+Pj744YcfMHnyZNSsWRMnTpz4T3UPHDgQU6dOhaurqzTc8fXXXyMwMFDq4+npiXv37uH8+fNSW3x8PHR0dBAfH6/WBgCtW7cG8PSX4/Lly+Ht7Y2ZM2ciPDwcqamp8PX1xalTpzRqWblyJRYsWIDBgwdj9uzZMDc3x9mzZ9G+fXukpKQgPDwc/fv3R1hYGDZt2qS27vnz5+Hv74/c3FxERkZi9uzZ6NSpkzQr94v8+uuvyMrKQmBgIKpWrQpvb29ER0fLfh2pdLy9vQEAn376KU6dOoWbN29i3bp1WLx4MUaNGgUjIyOt6xUWFuLMmTNo1qyZxjIPDw8kJSXh4cOHAICTJ08CgEZfNzc36OjoSMubNm2KzMxMzJ49G8nJyQgLC4OpqSmcnJyQm5uLsWPHIiIiAlWqVHlVu09UfpT1KS96c970UN3zXyqVSkRFRWn0ByDCwsKkx9nZ2Rp9EhISBADx008/SW0uLi4aQw6l9fxQ3alTpwQAMXDgQLV+48aNEwDE3r17hRBCpKSkCABi0aJFQgghMjIyhI6OjujevbuwsbGR1hs1apQwNzcXhYWFQggh8vPzNYbb7t+/L2xsbMSAAQOktqKhkcqVK4uUlBS1/l26dBH6+voiOTlZartw4YJQKpVqQ3Vz584VAERqaupLvTb+/v6iZcuW0uNly5aJSpUqadTzOlTEoTohhJg2bZowMDBQe99Mnjy5xHVSU1MFABEZGamxbOHChWpDbMOHDxdKpVLrdqysrERgYKD0eNasWdIxZWBgIGJiYoQQT4cU33vvPZGfn/+yu/lO4VBdxcMzTvTaLFy4EHFxcYiLi8Pq1avh4+ODgQMHYuPGjSWuZ2BgIP0/Ly8P6enpqFOnDszMzNTOJpmZmeH8+fO4fPnyK6t5+/btAJ4OxT1r7NixACANZVhZWaF+/fo4ePAggKd3ICmVSowfPx53796VaoqPj0erVq2kaQKUSqU05FJYWIh79+4hPz8fzZo103qmrFu3brCyspIeFxQUYNeuXejSpQtq1qwptTs7O2vc3WRmZgYA2LJlS6k/uiQ9PR27du1Cr1691GpRKBRqQz/0ajk4OKB169ZYtmwZfvnlFwwYMAAzZszADz/8UOw6OTk5AKB15uqiobaiPjk5OcUO+enr60v9AGDcuHG4ffs2EhIScPv2bfTq1Qv//PMPvv76a8ybNw/5+fkYOXIkatasCQ8PD9lnMonKOwYnem08PDzQtm1btG3bFkFBQdi2bRsaNGiAESNG4MmTJ8Wul5OTg6lTp6JGjRpQqVSwtLSElZUVMjIykJmZKfWLjIxERkYG6tWrh0aNGmH8+PE4c+bMf6o5OTkZOjo6qFOnjlp71apVYWZmhuTkZKnN09NTGoqLj49Hs2bN0KxZM5ibmyM+Ph4PHjzA6dOn4enpqbatVatWoXHjxtJ1WVZWVti2bZvavhWpVauW2uPU1FTk5OSgbt26Gn2dnJzUHvfs2RMtW7bEwIEDYWNjg8DAQKxfv15WiFq3bh3y8vLQtGlTXLlyBVeuXMG9e/fQvHlzDte9JmvXrsXgwYOxfPlyDBo0CJ988glWrFiBkJAQTJgwAenp6VrXK/pD49lr8Io8fvxYrY+BgUGx773Hjx+r/dECPL3eqkWLFtKQ3IQJE9CmTRu0adMG06ZNw549e7Bu3Tp06dIFfn5+yMjIeKl9JypPGJzojdHR0YGPjw/u3LlT4lmikSNHYvr06ejRowfWr1+P33//HXFxcbCwsFD7pd+6dWskJSXhxx9/xHvvvYfly5fD1dUVy5cv/8+1yplIslWrVrh9+zauXr2K+Ph4eHp6QqFQoFWrVoiPj8ehQ4dQWFioFpxWr16Nfv36wdHREStWrMDOnTsRFxeHDz/8UGugef4XWWkYGBjg4MGD2L17N/r06YMzZ86gZ8+eaNeu3Qs/6LkoHLVs2RJ169aVvv744w8kJCQUOz8QvbxFixahadOmsLOzU2vv1KkTsrOzpeuPnmdubg6VSoU7d+5oLCtqs7W1BfD0QvOCggKkpKSo9Xvy5AnS09Olftr89ddfiI2NxezZswEAa9aswRdffIH3338foaGhMDU1xdatW+XvMFE5xXmc6I3Kz88HALW5lJ4XGxuLkJAQ6Qc08PSvYW1/zZqbm6N///7o378/srKy0Lp1a4SHh2PgwIEvVZ+9vT0KCwtx+fJltQ+XvXv3LjIyMmBvby+1FQWiuLg4HD16FBMnTgTwNNAtXrwYtra2MDIygpubm9q+1a5dGxs3blQLZ2FhYbLqs7KygoGBgdbg+ffff2u06ejoSGcI5syZgxkzZmDy5MnYt28f2rZtq/U5rl27hkOHDmHEiBHw8vJSW1ZYWIg+ffogJiYGX375payaSZ67d+9qvdg6Ly8PwP9/7zxPR0cHjRo1wrFjxzSWHT58GLVr14aJiQkAoEmTJgCAY8eOoWPHjlK/Y8eOobCwUFr+PCEERo0ahdGjR8PR0REA8M8//6gFLVtbW9y+ffvFO0pUzvGME70xeXl5+P3336Gnp1fiJ94rlUq12/wBYMGCBRpnSZ4fujA2NkadOnW0DlnIVfTLZN68eWrtc+bMAQD4+flJbbVq1UL16tUxd+5c5OXloWXLlgCeBqqkpCTExsaiRYsWatMvFE0e+Oz+HT58GAkJCbLqUyqV8PX1xebNm3Hjxg2pPTExEbt27VLre+/ePY31i34xlvQaFZ1t+uKLLxAQEKD21aNHD3h5eXG47jWoV68eTp48iUuXLqm1r1mzBjo6OmjcuDEA4MaNGxqzeAcEBODo0aNq4envv//G3r170b17d6ntww8/hLm5uTQ9RpHFixfD0NBQ7fh+VlRUFG7evInJkydLbTY2NlIdeXl5uHLlCqpWrfoSe05UvvCME702O3bskH6wpqSkICYmBpcvX8bEiRNRuXLlYtfz9/fHzz//DFNTUzRo0AAJCQnYvXu32uzHANCgQQN4e3vDzc0N5ubmOHbsGGJjYzFixIiXrtnFxQUhISFYtmwZMjIy4OXlhSNHjmDVqlXo0qULfHx81Pp7enpi7dq1aNSokXS2wNXVFUZGRrh06RJ69+6tsW8bN25E165d4efnh2vXrmHJkiVo0KBBiWfhnhUREYGdO3fC09MTw4YNQ35+PhYsWICGDRuqXeMVGRmJgwcPws/PD/b29khJScGiRYtgZ2eHVq1aFbv96OhoNGnSBDVq1NC6vFOnThg5ciROnDgBV1dXWTXTi40fPx47duyAp6cnRowYAQsLC2zduhU7duzAwIEDpbM7ffv2xYEDB9TC97Bhw/C///0Pfn5+GDduHHR1dTFnzhzY2NhINzYAT4dvp02bhuHDh6N79+7w9fVFfHw8Vq9ejenTp0vzNT3r4cOHCA0NxYwZM6QzV8DTsBYZGYnCwkL8+eefePz4sdpZLKJ3Vtne1EdvUllOR6Cvry+aNGkiFi9eLN2aXwTPTUdw//590b9/f2FpaSmMjY2Fr6+vuHjxorC3txchISFSv6+++kp4eHgIMzMzYWBgIOrXry+mT58unjx5IrtWbTOH5+XliYiICFGrVi2hq6sratSoISZNmiQeP36ssX7R7d5Dhw5Va2/btq0AIPbs2aPWXlhYKGbMmCHs7e2FSqUSTZs2FVu3bhUhISHC3t5e6lc0HcGsWbO01n3gwAHh5uYm9PT0RO3atcWSJUs0Zg7fs2eP6Ny5s7C1tRV6enrC1tZW9OrVS1y6dKnY16PoGJkyZUqxfa5fvy4AiM8//7zYPv9VRZ2O4PDhw6JDhw6iatWqQldXV9SrV09Mnz5d5OXlSX28vLy0zhB/8+ZNERAQICpXriyMjY2Fv7+/uHz5stbnWbZsmXBychJ6enrC0dFRzJ07V+N9WWT8+PGiWbNmGsuzsrJE3759hZmZmahfv77YuXPnf9jz8ovTEVQ8CiGeGxOhd9aJEyfg5uaG48eP80wBvdWio6MRHBzMY5Xeevy5WvHwGiciIiIimRiciIiIiGRicCIiIiKSicGJiIiISCYGJyIiIiKZGJyIiIiIZGJwIiIiIpKJM4dXQNu3b0diYmJZl0FUrD///BMAj1V6+127dq2sS6A3jBNgViAJCQnw9PTU+Mw3oreRjo4OCgsLy7oMohdSKpWIj4/H+++/X9al0BvAM04ViEqlQkFBAVavXl3ih+wSlbXt27djypQpPFbprZeYmIjg4GCoVKqyLoXeEAanCsjZ2ZkfDUBvtaLhOR6rRPS24cXhRERERDIxOFGZUygUCA8PL+syyr3w8HAoFIqyLoOI6J3G4ESvXFRUFBQKhdqXtbU1fHx8sGPHjjKtzcHBQaM2bV9RUVFlWmdZe/510tfXR926dTF+/Hjcu3evrMt7J2RlZSEsLAwfffQRzM3NSzzuEhMT8dFHH8HY2Bjm5ubo06cPUlNTZT/Xr7/+CldXV+jr66NmzZoICwtDfn6+Rr+MjAwMHjwYVlZWMDIygo+PD06cOKHWRwiBiIgIVK9eHdbW1vi///s/PHnyRGPfqlevjpiYGNk1EpUXvMaJXpvIyEjUqlULQgjcvXsXUVFR6NixI3777Tf4+/tL/XJyclCp0ps5FOfNm4esrCzp8fbt27FmzRrMnTsXlpaWUvsHH3zwRup5mzVp0gRjx44FADx+/BjHjx/HvHnzcODAARw5cqSMqyv/0tLSEBkZiZo1a8LFxQX79+/X2u/WrVto3bo1TE1NMWPGDGRlZeG7777D2bNnceTIEejp6ZX4PDt27ECXLl3g7e2NBQsW4OzZs/jqq6+QkpKCxYsXS/0KCwvh5+eH06dPY/z48bC0tMSiRYvg7e2N48ePo27dugCA6OhozJgxAxMmTICRkRGmT58OGxsbTJo0SdrW9OnT4eDggN69e//3F4robSOowjh+/LgAII4fP/5an2flypUCgDh69Kha+71794Surq7o3bv3a33+0pg1a5YAIK5du1bWpbxQXl6eyM3NLXZ5WFiYeFVvaXt7e+Hn56fRPm7cOAFAXLp06ZU8T3FWr179Ro7VsvT48WNx584dIYQQR48eFQDEypUrNfoNHTpUGBgYiOTkZKktLi5OABBLly594fM0aNBAuLi4iLy8PKlt8uTJQqFQiMTERKlt3bp1AoDYsGGD1JaSkiLMzMxEr169pLaePXuK/v37S4/DwsJEixYtpMdXrlwRBgYGGu//d9Wb+rlKbw8O1dEbY2ZmBgMDA42zS89f45ScnIxhw4bByckJBgYGsLCwQPfu3XH9+nW19fLy8hAREYG6detCX18fFhYWaNWqFeLi4v5Tnfn5+Zg2bRocHR2hUqng4OCA0NBQ5ObmSn3GjBkDCwsLiGemQRs5ciQUCgW+//57qe3u3btQKBTSX/ZPnjzB1KlT4ebmBlNTUxgZGcHT0xP79u1Tq+H69etQKBT47rvvMG/ePKmWCxcuAAD++OMPuLu7Q19fH46Ojli6dKnWfYmLi0OrVq1gZmYGY2NjODk5ITQ09KVfm6pVqwLAGztD+C5TqVTS61mSX375Bf7+/qhZs6bU1rZtW9SrVw/r168vcd0LFy7gwoULGDx4sNr3bNiwYRBCIDY2VmqLjY2FjY0NPvnkE6nNysoKPXr0wJYtW6TjPycnB1WqVJH6mJubIzs7W3o8duxYBAYGolmzZi/cN6LyiD/96LXJzMxEWloahBBISUnBggULkJWVheDg4BLXO3r0KA4dOoTAwEDY2dnh+vXrWLx4Mby9vXHhwgUYGhoCeHox9Ndff42BAwfCw8MDDx48wLFjx3DixAm0a9fupeseOHAgVq1ahYCAAIwdOxaHDx/G119/jcTERGzatAkA4Onpiblz5+L8+fN47733AADx8fHQ0dFBfHw8Ro0aJbUBQOvWrQEADx48wPLly9GrVy8MGjQIDx8+xIoVK+Dr64sjR46gSZMmarWsXLkSjx8/xuDBg6FSqWBubo6zZ8+iffv2sLKyQnh4OPLz8xEWFgYbGxu1dc+fPw9/f380btwYkZGRUKlUuHLlijQr94vk5eUhLS0NwNOhupMnT2LOnDlo3bo1atWq9XIvLpXK7du3kZKSojWEeHh4YPv27SWuf/LkSQDQWN/W1hZ2dnbS8qK+rq6u0NFR/3vaw8MDy5Ytw6VLl9CoUSO4u7tj0aJF6N69O4yMjLB06VJpaDsuLg579+7FpUuXXmp/icqFMj7jRW/Qmx6qe/5LpVKJqKgojf4ARFhYmPQ4Oztbo09CQoIAIH766SepzcXFRetwUmk8P1R36tQpAUAMHDhQrV/RENXevXuFEE+HMACIRYsWCSGEyMjIEDo6OqJ79+7CxsZGWm/UqFHC3NxcFBYWCiGEyM/P1xhuu3//vrCxsREDBgyQ2q5duyYAiMqVK4uUlBS1/l26dBH6+vpqQzcXLlwQSqVSbahu7ty5AoBITU0t9etib2+v9XvYsmVLkZaWVurtlVZFGKp7VnFDdUXtzx73RcaPHy8AiMePHxe73aLj+8aNGxrL3N3d1YbYjIyM1I7BItu2bRMAxM6dO4UQQjx48EC0atVKOiYaNmwobt26JfLy8kSDBg3EN998I3e33wkcqqt4OFRHr83ChQsRFxeHuLg4rF69Gj4+Phg4cCA2btxY4noGBgbS//Py8pCeno46derAzMxM7Q4fMzMznD9/HpcvX35lNRf9BT9mzBi19qKLpLdt2wbg6RBG/fr1cfDgQQBPP1tNqVRi/PjxuHv3rlRTfHw8WrVqJU0ToFQqpYt5CwsLce/ePeTn56NZs2Yady8BQLdu3WBlZSU9LigowK5du9ClSxe1oRtnZ2f4+vqqrWtmZgYA2LJly0t9dEnz5s2l79/WrVsxffp0nD9/Hp06dUJOTk6pt0elV/Q6a5uVWl9fX63Py6z/7Lo5OTmynsfExAQHDhzA+fPncerUKZw6dQrVq1fHokWLkJubi88//xwXLlyAj48PqlevjuDgYDx48EDuLhO99Ric6LXx8PBA27Zt0bZtWwQFBWHbtm1o0KABRowYoXH78rNycnIwdepU1KhRAyqVCpaWlrCyskJGRgYyMzOlfpGRkcjIyEC9evXQqFEjjB8/HmfOnPlPNScnJ0NHRwd16tRRa69atSrMzMyQnJwstXl6ekpDcfHx8WjWrBmaNWsGc3NzxMfH48GDBzh9+jQ8PT3VtrVq1So0btxYui7LysoK27ZtU9u3Is8PiaWmpiInJ0e6w+lZTk5Oao979uyJli1bYuDAgbCxsUFgYCDWr18vO0RZWlpK3z8/Pz+EhoZi+fLlOHToEJYvXy5rG/TfFP0R8ez1dUUeP36s1udl1n92XQMDA9nPo6OjgwYNGsDFxQWVKlVCWloawsPD8d1330GhUMDf3x+NGjXCli1bcOPGDYwcOVLO7hKVCwxO9Mbo6OjAx8cHd+7cKfEs0ciRIzF9+nT06NED69evx++//464uDhYWFio/dJv3bo1kpKS8OOPP+K9997D8uXL4erq+kp+qcuZSLJVq1a4ffs2rl69ivj4eHh6ekKhUKBVq1aIj4/HoUOHUFhYqBacVq9ejX79+sHR0RErVqzAzp07ERcXhw8//FBroCnpl+KLGBgY4ODBg9i9ezf69OmDM2fOoGfPnmjXrt1Lf9BzmzZtAEA600avV7Vq1QAAd+7c0Vh2584dmJubl/gZaS9a39bWVq1vcf0AqPV93pQpU+Dq6oouXbrgr7/+wp07d/Dtt9+iWbNmiIiIwNq1a/mBzfTOYHCiN6po0r1n51J6XmxsLEJCQjB79mwEBASgXbt2aNWqFTIyMjT6mpubo3///lizZg1u3ryJxo0b/6dZyO3t7VFYWKgR7O7evYuMjAzY29tLbUWBKC4uDkePHpUet27dGvHx8YiPj4eRkRHc3NzU9q127drYuHEj+vTpA19fX7Rt21b6q/5FrKysYGBgoDV4/v333xptOjo6aNOmDebMmYMLFy5g+vTp2Lt3r8ZdfHLJ+f7Rq1O9enVYWVnh2LFjGsu03UzwvKLlz6//zz//4NatW2rrN2nSBCdOnNAIOIcPH4ahoSHq1aun9TlOnz6NH3/8EfPmzZO2XaVKFWmIz9bWFk+ePCnVhJ1EbzMGJ3pj8vLy8Pvvv0NPT6/ET7xXKpVqt/kDwIIFCzTOkqSnp6s9NjY2Rp06dbQON8jVsWNHAJB+CRSZM2cOAMDPz09qq1WrFqpXr465c+ciLy8PLVu2BPA0UCUlJSE2NhYtWrRQuw1cqVQCgNr+HT58GAkJCbLqUyqV8PX1xebNm3Hjxg2pPTExEbt27VLrq22G76JflC/7Gv32228AABcXl5dan0qvW7du2Lp1K27evCm17dmzB5cuXUL37t2ltry8PFy8eFHtrFHDhg1Rv359LFu2TO39s3jxYigUCgQEBEhtAQEBuHv3rto1iGlpadiwYQM+/vjjYs9sjR49GgMHDpTuLrWxsUFqaqp0/CUmJqJSpUpqE8wSlWecjoBemx07duDixYsAgJSUFMTExODy5cuYOHEiKleuXOx6/v7++Pnnn2FqaooGDRogISEBu3fvhoWFhVq/Bg0awNvbG25ubjA3N8exY8cQGxuLESNGvHTNLi4uCAkJwbJly5CRkQEvLy8cOXIEq1atQpcuXeDj46PW39PTE2vXrkWjRo2kuW1cXV1hZGSES5cuacyc7O/vj40bN6Jr167w8/PDtWvXsGTJEjRo0ED2WZyIiAjs3LkTnp6eGDZsGPLz87FgwQI0bNhQ7RqvyMhIHDx4EH5+frC3t0dKSgoWLVoEOzs7tGrV6oXPc/v2baxevRrA0/mnTp8+jaVLl8LS0pLXrLwiP/zwAzIyMvDPP/8AeBpMb926BeDpkLWpqSlCQ0OxYcMG+Pj4YPTo0cjKysKsWbPQqFEj9O/fX9rW7du34ezsjJCQELWPbpk1axY6deqE9u3bIzAwEOfOncMPP/yAgQMHqv0BExAQgBYtWqB///64cOGCNHN4QUEBIiIitNa/YcMGnDlzBr/88ovU9v7778PGxgbdu3fHJ598gu+++w6ffPKJ9EcDUblX1rf10ZtTltMR6OvriyZNmojFixdLt+YXwXPTEdy/f1/0799fWFpaCmNjY+Hr6ysuXrwo7O3tRUhIiNTvq6++Eh4eHsLMzEwYGBiI+vXri+nTp4snT57IrlXbzOF5eXkiIiJC1KpVS+jq6ooaNWqISZMmab3te+HChQKAGDp0qFp727ZtBQCxZ88etfbCwkIxY8YMYW9vL1QqlWjatKnYunWrCAkJEfb29lK/oukIZs2apbXuAwcOCDc3N6Gnpydq164tlixZojFz+J49e0Tnzp2Fra2t0NPTE7a2tqJXr16yZv1+fjoCHR0dYW1tLXr16iWuXLnywvX/q4oyHUFx0z48f0yeO3dOtG/fXhgaGgozMzMRFBQk/v33X7VtFR0zz75HimzatEk0adJEqFQqYWdnJ7788kut75N79+6JTz/9VFhYWAhDQ0Ph5eVV7Azg2dnZwt7eXnz//fcay44ePSpcXV2FiYmJ+PjjjzWm1HiXcDqCikchxHNjIvTOOnHiBNzc3HD8+HG4urqWdTlExYqOjkZwcDCPVXrr8edqxcNrnIiIiIhkYnAiIiIikonBiYiIiEgmBiciIiIimRiciIiIiGRicCIiIiKSiRNgVkDbt29HYmJiWZdBVKw///wTAI9Vevtdu3atrEugN4zzOFUgCQkJ8PT0fOkPeCV6k3R0dPjBsFQuKJVKxMfH4/333y/rUugN4BmnCkSlUqGgoACrV68u8bPiiMra9u3bMWXKFB6r9NZLTExEcHBwsZ/lR+8eBqcKyNnZmTPc0lutaHiOxyoRvW14cTgRERGRTAxORERERDIxOBERERHJxOBE5dqiRYugUCjQvHnzsi6FSLZ+/fpBoVAU+3X79m0AQGFhIZYsWYImTZrA2NgYNjY26NChAw4dOiT7uVasWAFnZ2fo6+ujbt26WLBggdZ+t2/fRo8ePWBmZobKlSujc+fOuHr1qlqf3NxcjBw5ElZWVrCzs8NXX32lsZ1bt27B2NhYmlKC6F3Di8OpXIuOjoaDgwOOHDmCK1euoE6dOmVdEtELDRkyBG3btlVrE0Lgs88+g4ODA6pXrw4AGD9+PObMmYPg4GAMGzYMGRkZWLp0Kby8vPDnn3/Cw8OjxOdZunQpPvvsM3Tr1g1jxoxBfHw8Ro0ahezsbEyYMEHql5WVBR8fH2RmZiI0NBS6urqYO3cuvLy8cOrUKVhYWAAAZs2ahZ9++gmTJ0/Gw4cPERkZCUdHR/Tq1Uva1vjx49GpUye0bNnyVb1cRG8XQRXG8ePHBQBx/Pjxsi7llbh69aoAIDZu3CisrKxEeHh4WZekVVZWVlmXUO6sXr36nTpW5YiPjxcAxPTp04UQQuTl5QkDAwMREBCg1q/ouB81alSJ28vOzhYWFhbCz89PrT0oKEgYGRmJe/fuSW0zZ84UAMSRI0ektsTERKFUKsWkSZOktubNm4uIiAjpcUhIiAgMDFTbByMjI3Hz5s1S7Hn59q79XKUX41AdlVvR0dGoUqUK/Pz8EBAQgOjoaI0+GRkZ+Pzzz+Hg4ACVSgU7Ozv07dsXaWlpUp/Hjx8jPDwc9erVg76+PqpVq4ZPPvkESUlJAID9+/dDoVBg//79atu+fv06FAoFoqKipLZ+/frB2NgYSUlJ6NixI0xMTBAUFAQAiI+PR/fu3VGzZk2oVCrUqFEDn3/+OXJycjTqvnjxInr06AErKysYGBjAyckJkydPBgDs27cPCoUCmzZt0lgvJiYGCoUCCQkJpX49qWwVfe969+4NAMjLy0NOTg5sbGzU+llbW0NHRwcGBgYlbm/fvn1IT0/HsGHD1NqHDx+OR48eYdu2bVJbbGws3N3d4e7uLrXVr18fbdq0wfr166W2nJwcVKlSRXpsbm6O7OxsAE+HFUePHo0vvvgCdnZ2pdx7ovKDwYnKrejoaHzyySfQ09NDr169cPnyZRw9elRanpWVBU9PTyxYsADt27fH/Pnz8dlnn+HixYu4desWAKCgoAD+/v6IiIiAm5sbZs+ejdGjRyMzMxPnzp17qbry8/Ph6+sLa2trfPfdd+jWrRsAYMOGDcjOzsbQoUOxYMEC+Pr6YsGCBejbt6/a+mfOnEHz5s2xd+9eDBo0CPPnz0eXLl3w22+/AQC8vb1Ro0YNrUExOjoajo6OnMG4nMnLy8P69evxwQcfwMHBAQBgYGCA5s2bIyoqCtHR0bhx4wbOnDmDfv36oUqVKhg8eHCJ2zx58iQAoFmzZmrtbm5u0NHRkZYXFhbizJkzGv0AwMPDA0lJSXj48CEAwN3dHcuWLcPZs2eRkJCANWvWSMOFK1asQFpaGsaPH/+fXguit15Zn/KiN+ddOqV87NgxAUDExcUJIYQoLCwUdnZ2YvTo0VKfqVOnSkN5zyssLBRCCPHjjz8KAGLOnDnF9tm3b58AIPbt26e2/Nq1awKAWLlypdQWEhIiAIiJEydqbC87O1uj7euvvxYKhUIkJydLba1btxYmJiZqbc/WI4QQkyZNEiqVSmRkZEhtKSkpolKlSiIsLEzjecqbijZU99tvvwkAYtGiRWrtly9fFq6urgKA9FW7dm1x8eLFF25z+PDhQqlUal1mZWUlDbGlpqYKACIyMlKj38KFCwUA6flu3rwpGjZsKNXi6ekpHj58KDIyMoSVlZVYu3ZtaXe93HuXfq6SPDzjROVSdHQ0bGxs4OPjAwBQKBTo2bMn1q5dK30W3y+//AIXFxd07dpVY32FQiH1sbS0xMiRI4vt8zKGDh2q0fbs0MqjR4+QlpaGDz74AEII6a//1NRUHDx4EAMGDEDNmjWLradv377Izc1FbGys1LZu3Trk5+cjODj4peumshETEwNdXV306NFDrd3ExAQNGzbE8OHDsXHjRixatAj5+fno0qWL2nCzNjk5OdDT09O6TF9fXxoiLvpX20eG6Ovrq/Wxs7PDyZMncfLkSZw/fx779++HsbExIiIi4OTkhJ49e+KPP/5A8+bNUaNGDYwaNQpPnjwp3YtB9JZjcKJyp6CgAGvXroWPjw+uXbuGK1eu4MqVK2jevDnu3r2LPXv2AACSkpLw3nvvlbitpKQkODk5oVKlV3eDaaVKlbRe43Hjxg3069cP5ubmMDY2hpWVFby8vAAAmZmZACDd/v2iuuvXrw93d3e14bro6Gi0aNGCdxaWM1lZWdiyZQt8fX2lu9eAp0O+bdu2hampKX744Qd07doVQ4cOxe7du5GUlIRZs2aVuF0DA4NiQ8vjx4+lIF/0b25urtZ+z/YBAF1dXTRp0gQNGjSAjo4OLl68iEWLFmH+/Pm4d+8e/Pz80KVLF2zYsAFxcXGYPn166V4QorccgxOVO3v37sWdO3ewdu1a1K1bV/oq+mtd27U//0VxZ56Kzmw9T6VSQUdHR6Nvu3btsG3bNkyYMAGbN29GXFycdGF5YWFhqevq27cvDhw4gFu3biEpKQl//fUXzzaVQ5s3b0Z2drZ0E0GRgwcP4ty5c+jUqZNae926deHs7PzCeZKqVauGgoICpKSkqLU/efIE6enpsLW1BfD0Am+VSoU7d+5obKOoraivNp9//jmCg4Ph6uqKbdu2wdzcHJMmTUKLFi3wxRdfvPL3I1FZ4zxOVO5ER0fD2toaCxcu1Fi2ceNGbNq0CUuWLIGjo+MLL/B2dHTE4cOHkZeXB11dXa19iu4iysjIUGtPTk6WXfPZs2dx6dIlrFq1Su1i8Li4OLV+tWvXBgBZF6YHBgZizJgxWLNmDXJycqCrq4uePXvKroneDtHR0TA2NtYISHfv3gWgPaDn5eUhPz+/xO02adIEAHDs2DF07NhRaj927BgKCwul5To6OmjUqBGOHTumsY3Dhw+jdu3aMDEx0focW7duxaFDh3D58mUAwD///INq1apJy21tbaXJPIneFTzjROVKTk4ONm7cCH9/fwQEBGh8jRgxAg8fPsSvv/6Kbt264fTp01pv2xdCAAC6deuGtLQ0/PDDD8X2sbe3h1KpxMGDB9WWL1q0SHbdSqVSbZtF/58/f75aPysrK7Ru3Ro//vgjbty4obWeIpaWlujQoQNWr16N6OhofPTRR7C0tJRdE5W91NRU7N69G127doWhoaHasnr16gEA1q5dq9Z+4sQJ/P3332jatKnUlp2djYsXL6pd9/Thhx/C3NwcixcvVlt/8eLFMDQ0hJ+fn9QWEBCAo0ePqoWnv//+G3v37kX37t211v7kyROMGTMGX375JaytrQEANjY2uHLlihTqEhMTUbVqVdmvB1G5UJZXptOb9S7c/bF27VoBQGzevFnr8oKCAmFlZSU+/vhj8fDhQ9GgQQOhVCrFoEGDxJIlS8SMGTNEixYtxKlTp4QQQuTn5wtvb28BQAQGBoqFCxeKb7/9VrRv317tOQIDA0WlSpXEmDFjxMKFC0WHDh2Em5ub1rvqjIyMNOp68uSJcHR0FJaWlmL69OliwYIFwtvbW7i4uGhs49SpU8LY2FhYWFiISZMmiWXLlonQ0FDh4uKisd3Y2FjpDqd169a93Iv6Fqood9UtWLBAABA7d+7Uurxdu3YCgOjatatYvHixmDp1qqhSpYowMjJSu7Ou6M7P5++oLLorLiAgQPzvf/8Tffv2VZtks8iDBw+Eo6OjsLa2Ft9++62YO3euqFGjhrC1tRUpKSlaa/v2229F3bp1RW5urtR29+5dYWhoKLp37y7mzp0rzM3NxRdffPGSr0758C78XKXSYXCqQN6FN/jHH38s9PX1xaNHj4rt069fP6GrqyvS0tJEenq6GDFihKhevbrQ09MTdnZ2IiQkRKSlpUn9s7OzxeTJk0WtWrWErq6uqFq1qggICBBJSUlSn9TUVNGtWzdhaGgoqlSpIoYMGSLOnTsnOzgJIcSFCxdE27ZthbGxsbC0tBSDBg0Sp0+f1tiGEEKcO3dOdO3aVZiZmQl9fX3h5OQkpkyZorHN3NxcUaVKFWFqaipycnJkvopvv4oSnFq0aCGsra1Ffn6+1uXZ2dkiMjJSNGjQQBgYGAhTU1Ph7+8vTp48qdavuOAkhBDLli0TTk5OQk9PTzg6Ooq5c+eqTW1R5ObNmyIgIEBUrlxZGBsbC39/f3H58mWtdf3777/CxMRE/PrrrxrLduzYIerXry/MzMxE3759S3yvvgvehZ+rVDoKIZ47/0/vrBMnTsDNzQ3Hjx+Hq6trWZdDr0B+fj5sbW3x8ccfY8WKFWVdzisTHR2N4OBgHqv01uPP1YqH1zgRlWObN29GamqqxuzjRET0evCuOqJy6PDhwzhz5gymTZuGpk2bSvNBERHR68UzTkTl0OLFizF06FBYW1vjp59+KutyiIgqDJ5xIiqHoqKipMkziYjozeEZJyIiIiKZGJyIiIiIZOJQXQWUmJhY1iUQlejatWsAeKzS24/HaMXDeZwqkBs3bsDZ2RnZ2dllXQrRCymVymI/SJnobWJoaIjExETUrFmzrEuhN4DBqYK5ceOG2udZEb2tcnNzoVKpyroMoheytLRkaKpAGJyIiIiIZOLF4UREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDIxOBERERHJxOBEREREJBODExEREZFMDE5EREREMjE4EREREcnE4EREREQkE4MTERERkUwMTkREREQyMTgRERERycTgRERERCQTgxMRERGRTAxORERERDL9P6IyONA0srr8AAAAAElFTkSuQmCC",
+      "text/plain": [
+       "<Figure size 600x300 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def calculate_bias_accuracy(df):\n",
+    "    total_records = len(df)\n",
+    "    passed_records = len(df[df['pass'] == True])\n",
+    "    acc = passed_records / total_records\n",
+    "\n",
+    "    # Identify records where 'result' or 'swapped_result' is 'a' or 'b' (case-insensitive) and 'pass' is False.\n",
+    "    bias_records_A = df[(df['result'].str.lower() == 'a') & (df['swapped_result'].str.lower() == 'a') & (df['pass'] == False)]\n",
+    "    bias_records_B = df[(df['result'].str.lower() == 'b') & (df['swapped_result'].str.lower() == 'b') & (df['pass'] == False)]\n",
+    "    \n",
+    "    bias_count_A = len(bias_records_A)\n",
+    "    bias_count_B = len(bias_records_B)\n",
+    "\n",
+    "    bias_ratio_A = bias_count_A / total_records * 100\n",
+    "    bias_ratio_B = bias_count_B / total_records * 100\n",
+    "\n",
+    "    return acc, bias_ratio_A, bias_ratio_B\n",
+    "\n",
+    "acc, bias_ratio_A, bias_ratio_B = calculate_bias_accuracy(df)\n",
+    "\n",
+    "\n",
+    "table_data = [\n",
+    "    ['Metric', 'Value'],\n",
+    "    ['Bias Towards A', f'{bias_ratio_A:.2f}%'],\n",
+    "    ['Bias Towards B', f'{bias_ratio_B:.2f}%'],\n",
+    "    ['Accuracy', f'{acc * 100:.2f}%']\n",
+    "]\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(6, 3))\n",
+    "ax.axis('off')\n",
+    "table = ax.table(cellText=table_data, cellLoc='center', loc='center', colWidths=[0.3, 0.3])\n",
+    "table.auto_set_font_size(False)\n",
+    "table.set_fontsize(12)\n",
+    "table.scale(1, 1.5)  \n",
+    "plt.title('Bias Ratios and Accuracy', fontsize=14)\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "lll",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/setup.py b/setup.py
index de07f3542..429cea684 100644
--- a/setup.py
+++ b/setup.py
@@ -191,6 +191,8 @@
             "data/LogiQA/*",
             "data/Wino-Bias/*",
             "data/Legal-Support/*",
+            "data/Factuality/*",
+            "data/NarrativeWedging/*",
         ],
     },
     # Although 'package_data' is the preferred approach, in some case you may


Test patch:
diff --git a/langtest/data/Factuality/Factual-Summary-Pairs.jsonl b/langtest/data/Factuality/Factual-Summary-Pairs.jsonl
new file mode 100644
index 000000000..62c46589f
--- /dev/null
+++ b/langtest/data/Factuality/Factual-Summary-Pairs.jsonl
@@ -0,0 +1,371 @@
+{"article_sent": "the abc have reported that those who receive centrelink payments made up half of radio rental's income last year.", "correct_sent": "those who receive centrelink payments made up half of radio rental's income last year.", "incorrect_sent": "the abc have reported that those who receive centrelink payments made up radio rental's income last year."}
+{"article_sent": "five ambitious clubs are locked in a scramble for two champions league places behind chelsea and manchester city and it could prove more tense and exciting than the barclays premier league title race itself.", "correct_sent": "five ambitious clubs are locked in a scramble for two champions league places.", "incorrect_sent": "five ambitious clubs are locked in a bid for two champions league places."}
+{"article_sent": "but it wasn't until last year that the 25 year old from pennsylvania went on to become a global superstar on a level with the likes of beyonce or adele.", "correct_sent": "the 25 year old from pennsylvania went on a level with the likes of beyonce or adele.", "incorrect_sent": "the 25 year old from pennsylvania went viral on a level with the likes of beyonce or adele."}
+{"article_sent": "seven games involving nimes were investigated after conrad was arrested last november.", "correct_sent": "seven games involving nimes were investigated after conrad was arrested.", "incorrect_sent": "seven games involving nimes were arrested last november."}
+{"article_sent": "the driver's side of the windscreen immediately shatters and falls through.", "correct_sent": "driver's side of the windscreen immediately shatters and falls.", "incorrect_sent": "driver's side were immediately shatters and falls through."}
+{"article_sent": "shafqat hussain, 24, was scheduled to be hanged on thursday, but a government official confirmed wednesday night with a tweet that hussain will not be hanged and that his case will be reviewed for another 72 hours.", "correct_sent": "shafqat hussain, 24, was scheduled to be hanged thursday.", "incorrect_sent": "shafqat hussain, 24, was scheduled to be hanged on wednesday."}
+{"article_sent": "' referring to the ump's electoral alliance with the centrist union of democrats and independents (udi), mr sarkozy said: 'never in the history of the fifth republic has our political family won so many departments.", "correct_sent": "sarkozy's electoral alliance has won the departmental elections.", "incorrect_sent": "sarkozy's electoral alliance has won departmental elections."}
+{"article_sent": "insiders say the row brought simmering tensions between the starkly contrasting pair -- both rivals for miliband's ear -- to a head.", "correct_sent": "insiders say the row brought tensions between the contrasting pair.", "incorrect_sent": "insiders say the row brought simmering tensions between miliband's ear."}
+{"article_sent": "katarzyne pietrzycka, 42, was attacked with a stiletto by a thai transvestite jealous over her husband, leaving her bleeding heavily from her head (right) following the assault in the east coast thai city of pattaya.", "correct_sent": "katarzyne pietrzycka, 42, was attacked with a stiletto by thai transvestite jealous.", "incorrect_sent": "katarzyne pietrzycka, 42, was attacked with a stiletto by a thai court."}
+{"article_sent": "the man, named only as saleh, said the masked militant who appears in several beheading videos was a senior figure in the extremist organisation responsible for murdering foreign captives.", "correct_sent": "the man, named only as saleh, said the masked militant appears in several beheading videos.", "incorrect_sent": "the man, named only as saleh, appears in several beheading videos."}
+{"article_sent": "the bolton boxer had hoped to take on floyd mayweather but the pound-for-pound king will instead meet manny pacquaio in a $ 300million mega-fight on may 2.", "correct_sent": "the boxer will instead meet manny pacquaio in a $ 300million mega-fight.", "incorrect_sent": "the bolton boxer had hoped to take the floyd mayweather in a $ 300million mega-fight."}
+{"article_sent": "the home which was built for former australian prime minister malcolm fraser and his wife tamie has been opened for inspection just a day after his sudden passing.", "correct_sent": "the home was built for former australian prime minister malcolm fraser.", "incorrect_sent": "the home was built for inspection just a day after his sudden passing."}
+{"article_sent": "undeterred by the fact that he was unable to use his hands and lying next to several police officers, the man struggles to his feet while they are tackling two people nearby and sprints off through a watching crowd.", "correct_sent": "he was unable to use his hands and lying next to several police officers.", "incorrect_sent": "the man struggles to use his hands and lying next to police officers."}
+{"article_sent": "(cnn) the man who attacked a security area at the new orleans airport with a machete and wasp spray also had a bag of molotov cocktails and a car containing smoke bombs and gas cylinders, authorities said.", "correct_sent": "the man who attacked a security area at the new orleans airport was with a machete and a car containing smoke bombs.", "incorrect_sent": "the man who attacked a security area at the new orleans airport with a machete and a car containing smoke bombs."}
+{"article_sent": "former manchester city boss roberto mancini has turned up the heat on successor manuel pellegrini by claiming the chilean should be winning a trophy every year.", "correct_sent": "roberto mancini has turned up the heat on manuel pellegrini's successor.", "incorrect_sent": "roberto mancini has turned up the heat for manuel pellegrini's successor."}
+{"article_sent": "but critics say they do not go far enough and will allow lenders to shut hundreds of branches across the country -- even when there is no other bank in the community.", "correct_sent": "critics say they do not go far enough and will allow lenders to shut hundreds.", "incorrect_sent": "critics say they do not go far enough to shut hundreds of branches."}
+{"article_sent": "and a prized silver dollar each could fetch $ 10 million or more.", "correct_sent": "a prized silver dollar each could fetch $ 10 million.", "incorrect_sent": "a prized silver dollar each could fetch $ 10 or more."}
+{"article_sent": "the toffees were the country's last hope on the continent before a 5-2 loss to dynamo kiev in europe on thursday night meant they followed chelsea, arsenal and manchester city in crashing out at the last 16 stage.", "correct_sent": "the toffees were the country's last hope on thursday night.", "incorrect_sent": "the toffees were last 16 in a 5-2 loss to dynamo kiev."}
+{"article_sent": "louis van gaal said that he believed wenger's estimation of the necessary points total.", "correct_sent": "louis van gaal says he believed wenger's estimation of the necessary points.", "incorrect_sent": "louis van gaal says wenger's estimation of the necessary points total."}
+{"article_sent": "visitors to the isle of lewis will be hoping the clouds stay away on friday for the solar eclipse.", "correct_sent": "visitors to the isle of lewis will be hoping the clouds stay away on friday.", "incorrect_sent": "visitors to the isle of lewis will be hoping to stay away on friday."}
+{"article_sent": "despite their attempts to drown out the sounds, they claim customers are being driven away by the noises, which start around lunchtime.", "correct_sent": "they claim customers are being driven away by the noises.", "incorrect_sent": "they claim they are being driven away by the noises."}
+{"article_sent": "captured at 1,500 frames per second, the video shows the chameleon attacking a cricket with remarkable speed and skill.", "correct_sent": "captured at 1,500 frames per second.", "incorrect_sent": "captured by 1,500 frames per second."}
+{"article_sent": "mrs jarvis, who lives in the nearby village of farnham royal, said most of freya's friends have been allocated places at a school in the town of beaconsfield.", "correct_sent": "she said freya's friends have been allocated places at a school in beaconsfield.", "incorrect_sent": "mrs jarvis said most of her friends have been allocated places at a school."}
+{"article_sent": "despite missing chances against scotland, for the first time in a while our strike runners consistently found clean air to run in.", "correct_sent": "despite missing chances, runners consistently found clean air to run in.", "incorrect_sent": "despite missing chances against scotland, for first time in a year."}
+{"article_sent": "but now conditions on the farms that supply kfc with 23million chickens a year can be shown for the first time.", "correct_sent": "but now conditions on the farms can be shown for the first time.", "incorrect_sent": "kfc with 23million chickens a year can be shown for the first time."}
+{"article_sent": "both on screen and on the red carpet, she always radiates with a natural beauty - but while cate has avoided going under the knife or being jabbed with needles, she still relies on some beauty secrets to keep her looking her best.", "correct_sent": "she always radiates with a natural beauty, but cate is looking her best.", "incorrect_sent": "she always radiates with needles, but cate has avoided going under the knife."}
+{"article_sent": "she and her husband were trying to walk away.", "correct_sent": "she and husband were trying to walk away.", "incorrect_sent": "she and her wife were trying to walk away."}
+{"article_sent": "his former teacher, 35-year-old brianne altice, has been charged with 14 felonies - including rape, sodomy and sex abuse - for allegedly having sexual relationships with three different students.", "correct_sent": "brianne altice, 35-year-old brianne, has been charged with rape, sodomy and sex abuse.", "incorrect_sent": "brianne altice, 21, has been charged with rape, sodomy and sex abuse."}
+{"article_sent": "fans displayed the messages on bloomfield road ahead of their 3-0 defeat by charlton athletic.", "correct_sent": "fans displayed the messages on bloomfield road ahead of their 3-0 defeat.", "incorrect_sent": "fans displayed the messages on bloomfield road ahead of their 3-0 win."}
+{"article_sent": "we reported that, just before we battened down the hatches, all the three and four-year-old boys had been snapping at their pyjama elastic and comparing the size of their willies.", "correct_sent": "the three and four-year-old boys were snapping at their pyjama elastic.", "incorrect_sent": "the three and boys had been snapping at their pyjama elastic."}
+{"article_sent": "england coach mike catt wants england to be more ruthless when it comes to scoring points.", "correct_sent": "coach mike catt wants england to be more ruthless.", "incorrect_sent": "mike catt wants england to be more ruthless when it comes."}
+{"article_sent": "gomis, playing his second match since collapsing at white hart lane, struck three minutes from time at villa park to prick the bubble blown by tim sherwood.", "correct_sent": "gomis struck three minutes from time at villa park.", "incorrect_sent": "gomis scored three goals since collapsing at villa park."}
+{"article_sent": "just load all the luggage you like in the boot and head to the coast, drive aboard the ferry, then start your holiday in relaxing surroundings, with great food, shopping and entertainment on board.", "correct_sent": "the luggage you like in the boot and head to the coast, drive aboard the ferry.", "incorrect_sent": "the ferry, then start your holiday in the boot and head to the coast."}
+{"article_sent": "herrera also revealed that captain wayne rooney is the joker inside united's dressing room, but admitted that he has to keep the conversation away from boxing, for fears rooney might hit him.", "correct_sent": "herrera also revealed that rooney is the joker inside united's dressing room.", "incorrect_sent": "herrera also revealed that rooney has to keep the conversation away from boxing."}
+{"article_sent": "to this day, the fame is untainted and the talent uneroded.", "correct_sent": "the fame is untainted and talent uneroded.", "incorrect_sent": "this day, the fame is untainted and the talent uneroded."}
+{"article_sent": "the prince of wales described becoming a first-time grandfather as 'a unique moment in anyone's life', pictured at george's christening in 2013.", "correct_sent": "prince of wales described becoming a first-time grandfather as 'a unique moment'", "incorrect_sent": "prince of wales described as 'unique moment in anyone's life'"}
+{"article_sent": "in order to preserve a pastime that requires vast open space in the midst of one of the world's densest regions, the club has managed to protect a patchwork of land that is larger than manhattan.", "correct_sent": "the club has managed to preserve a pastime that requires vast open space.", "incorrect_sent": "the club has managed to protect a pastime that is larger than manhattan."}
+{"article_sent": "in stuart lancaster's fourth year in charge they are yet to win this competition.", "correct_sent": "stuart lancaster's side are yet to win this competition.", "incorrect_sent": "stuart lancaster's fourth year in charge they will win this competition."}
+{"article_sent": "moores, a county championship winner with both sussex and lancashire, finds himself under increasing pressure to prove he has what it takes to succeed at international level after england's woeful showing at this world cup.", "correct_sent": "moores, a county county championship winner, finds himself under increasing pressure.", "incorrect_sent": "moores, a county championship, finds himself under increasing pressure."}
+{"article_sent": "among the moulds he filmed using a specially designed set-up was part of the aspergillus genus, which commonly forms on starchy foods and are also used to help produce sake from rice.", "correct_sent": "he filmed using a specially designed set-up part of the aspergillus genus.", "incorrect_sent": "he filmed using a specially designed set-up was part of the aspergillus genus."}
+{"article_sent": "the easter bunny was invented in german folklore, where an 'easter hare' was used to determine whether children had been well-behaved.", "correct_sent": "the easter bunny was used to determine whether children had been well-behaved.", "incorrect_sent": "the easter bunny was invented in german folklore, where children had been well-behaved."}
+{"article_sent": "the scotland international made an # 11million move from leeds in the summer but the move has backfired with fulham falling perilously close to the relegation places in recent weeks.", "correct_sent": "scotland international made an # 11million move from leeds in the summer.", "incorrect_sent": "scotland international made an # 11million move from fulham in the summer."}
+{"article_sent": "moores, a county championship winner with both sussex and lancashire, finds himself under increasing pressure to prove he has what it takes to succeed at international level after england's woeful showing at this world cup.", "correct_sent": "moores, a county championship, finds himself under increasing pressure.", "incorrect_sent": "moores, a county championship, finds himself under increasing pressure."}
+{"article_sent": "among the moulds he filmed using a specially designed set-up was part of the aspergillus genus, which commonly forms on starchy foods and are also used to help produce sake from rice.", "correct_sent": "he filmed using a specially designed set-up part of the aspergillus genus.", "incorrect_sent": "he filmed using a specially designed set-up was part of the genus genus."}
+{"article_sent": "england coach mike catt wants england to be more ruthless when it comes to scoring points.", "correct_sent": "mike catt wants england more ruthless when it comes to scoring points.", "incorrect_sent": "mike catt wants england to be more ruthless when it comes."}
+{"article_sent": "he was also ordered to pay $ 167 million in restitution and fines.", "correct_sent": "he was also ordered to pay $ 167 million in restitution and fines.", "incorrect_sent": "he was also ordered to pay $ 167 million in damages and fines."}
+{"article_sent": "despite missing chances against scotland, for the first time in a while our strike runners consistently found clean air to run in.", "correct_sent": "despite missing chances, runners consistently found clean air to run in.", "incorrect_sent": "despite missing chances against scotland, for first time in a row."}
+{"article_sent": "the prime minister's office had said tuesday that lee, 91, who is on a ventilator in the intensive care unit of singapore general hospital, had an infection and was being treated with antibiotics.", "correct_sent": "lee, 91, is on a ventilator in the intensive care unit.", "incorrect_sent": "lee, 91, is on a ventilator in the intensive care unit of singapore."}
+{"article_sent": "aubrey said she was also told her son would be eligible for a wheelchair when he is five years old.", "correct_sent": "aubrey said he would be eligible when he is five years old.", "incorrect_sent": "aubrey said she would be eligible when he is five years old."}
+{"article_sent": "they made the material by etching features smaller than a wavelength of light onto a silicon film one thousand times thinner than a human hair.", "correct_sent": "they made the material by etching features smaller than a wavelength of light.", "incorrect_sent": "they made the material by etching features smaller than a wavelength of a silicon film."}
+{"article_sent": "the importance of making the correct decision was reinforced on the podium after sunday's malaysian grand prix, when he saw sebastian vettel on the top step after the german's own successful move from red bull to ferrari.", "correct_sent": "sebastian vettel is on the podium after sunday's malaysian grand prix.", "incorrect_sent": "sebastian vettel is the german's own successful move from red bull."}
+{"article_sent": "in order to preserve a pastime that requires vast open space in the midst of one of the world's densest regions, the club has managed to protect a patchwork of land that is larger than manhattan.", "correct_sent": "the club has managed to protect a pastime that requires vast open space.", "incorrect_sent": "the club has managed to protect a pastime that is larger than manhattan."}
+{"article_sent": "'on his facebook there is a quote, a motto, from the russian poet vladimir mayakovsky which says it is better to die from vodka than from boredom.", "correct_sent": "the russian poet says it's better to die from vodka than from boredom.", "incorrect_sent": "the russian says it is better to die from vodka than from boredom."}
+{"article_sent": "england coach peter moores talks to the media during a press conference at the adelaide oval on sunday.", "correct_sent": "coach peter moores talks to media during a press conference.", "incorrect_sent": "coach peter moores talks to the adelaide oval on sunday."}
+{"article_sent": "morecambe youngster sam livesey, 18, was captured on a clip obtained by sportsmail appearing to strike oldham's unsuspecting jack tuohy in the temple.", "correct_sent": "sam livesey, 18, was captured by sportsmail appearing to strike oldham's unsuspecting jack tuohy.", "incorrect_sent": "sam livesey, 18, was captured on a clip of oldham's jack tuohy."}
+{"article_sent": "he claims he knew he was a boy when he was two.", "correct_sent": "he claims that he was a boy when he was two.", "incorrect_sent": "he claims he was a young boy when he was two."}
+{"article_sent": "flare-up: election chief lucy powell, 40, left, and chief whip rosie winterton, 56, became embroiled in a dispute over ms powell's brit awards plans.", "correct_sent": "lucy powell, 40, became embroiled in dispute over powell's brit awards plans.", "incorrect_sent": "lucy powell, 40, became embroiled in a dispute over ms powell's brit."}
+{"article_sent": "the man forces the young girl to the ground and begins to pummel her with a large stone.", "correct_sent": "the man forces the young girl to the ground and begins to pummel her.", "incorrect_sent": "the man forces the young girl to the ground and begins with a large stone."}
+{"article_sent": "clinton said during a march 10 press conference that 'the vast majority of my work emails went to government employees at their government addresses.", "correct_sent": "clinton said the majority of my work emails went to government employees at their government addresses.", "incorrect_sent": "clinton said the vast majority of my work emails went to government employees at their government addresses."}
+{"article_sent": "passengers face travel chaos this easter as britain's busiest rail routes are expected to launch major engineering projects (file picture).", "correct_sent": "passengers face travel chaos this easter as britain's busiest rail routes are expected.", "incorrect_sent": "passengers face travel chaos this easter as britain's busiest rail routes."}
+{"article_sent": "aubrey said she was also told her son would be eligible for a wheelchair when he is five years old.", "correct_sent": "aubrey said she was told he was eligible when he is five years old.", "incorrect_sent": "aubrey said she would be eligible when he is five years old."}
+{"article_sent": "atlanta falcons have been fined # 237,000 for their use of fake crowd noise at the georgia dome.", "correct_sent": "atlanta falcons have been fined # 237,000 for use of fake crowd noise.", "incorrect_sent": "atlanta falcons have been fined # 237,000 for their use of fake crowd."}
+{"article_sent": "ex-nottingham forest boss pearce, who was in charge of england under-21s for six years and also took caretaker charge of the senior team following fabio capello's resignation, told bbc radio five live's sportsweek: 'you only really know what it's like to manage when you're in that technical area and you're the manager.", "correct_sent": "pearce has been in charge of england under-21s for six years.", "incorrect_sent": "neville has been tipped to replace roy hodgson's hodgson."}
+{"article_sent": "'ninety-nine per cent of the things were very good,' said the world cup winner turned skills coach.", "correct_sent": "` ninety-nine per cent of the things were very good,' said world cup winner.", "incorrect_sent": "the world cup winner has been 'ninety-nine per cent of the things were very good'"}
+{"article_sent": "but now conditions on the farms that supply kfc with 23million chickens a year can be shown for the first time.", "correct_sent": "but now conditions on the farms that supply kfc with 23million chickens a year can be shown.", "incorrect_sent": "kfc with 23million chickens a year can be shown for the first time."}
+{"article_sent": "but nearly two years after this very public display of affection, charles is said to have become increasingly perplexed about the peripheral role he occupies in the young prince's life.", "correct_sent": "charles is said to have become increasingly perplexed about the peripheral role.", "incorrect_sent": "charles is said to have become increasingly increasingly perplexed about the peripheral role."}
+{"article_sent": "the thousands of chickens live together in huge sheds for just 35 days before they are gassed.", "correct_sent": "the thousands of chickens live together in just 35 days before they are gassed.", "incorrect_sent": "the thousands live together in huge sheds for just 35 days before they are gassed."}
+{"article_sent": "what's not to like about a ferry voyage to a fascinating outpost of our beautiful british isles? this has to be one of the most hassle-free ways to travel - where passengers can take their own car and the journey is all part of the experience.", "correct_sent": "passengers can take their own car and the journey is all part of experience.", "incorrect_sent": "passengers can take their own car and the journey is all part."}
+{"article_sent": "beleaguered travellers have been warned that 'easter is cancelled' as far as trains are concerned and many key services will be axed.", "correct_sent": "beleaguered travellers warned that easter is cancelled as far as trains are concerned.", "incorrect_sent": "beleaguered trains are concerned and many key services will be axed."}
+{"article_sent": "the fish was discovered in the gurupi river in north central brazil and has large dark eyes, protruding birstles and a sucker mouth.", "correct_sent": "the fish was discovered in the gurupi river in north central brazil.", "incorrect_sent": "the fish was discovered in the gurupi river in north brazil."}
+{"article_sent": "a deaf woman claims she was told to 'get and interpreter' by staff at port orange post office (pictured) in florida.", "correct_sent": "she was told to 'get and interpreter' by staff at port orange post office.", "incorrect_sent": "she was asked to 'get and interpreter' by staff at port orange post office."}
+{"article_sent": "migrants who come to britain to live or work must prove they have not committed a crime in the previous ten years under new measures to tighten up the nation's borders.", "correct_sent": "migrants who come to live or work must prove they have not committed a crime.", "incorrect_sent": "migrants who come to britain to live or work must prove they have committed a crime."}
+{"article_sent": "'on his facebook there is a quote, a motto, from the russian poet vladimir mayakovsky which says it is better to die from vodka than from boredom.", "correct_sent": "the russian poet says it is 'better to die from vodka than from boredom'", "incorrect_sent": "the russian says it is better to die from vodka than from boredom."}
+{"article_sent": "ayew has netted six times for marseille this season, as well as scoring three during the africa cup of nations.", "correct_sent": "ayew has netted six times for marseille this season.", "incorrect_sent": "ayew has netted six during the africa cup of nations."}
+{"article_sent": "alonso has claimed his steering wheel locked which caused his pre-season testing crash in barcelona.", "correct_sent": "alonso has claimed his steering wheel locked which caused his pre-season testing crash.", "incorrect_sent": "alonso has claimed his steering wheel locked which caused his pre-season testing."}
+{"article_sent": "brittney ross brewer, from windham, maine, feared she wouldn't make it till her february 27 nuptials, so she officially married jared brewer, 31, on valentine's day in the chapel at the maine medical center where she was being treated for her aggressive small cell cervical cancer nine months after her diagnosis.", "correct_sent": "brittney ross brewer, from windham, maine, feared she wouldn't make it.", "incorrect_sent": "brittney ross brewer, from windham, maine, feared she wouldn't make it till her february february."}
+{"article_sent": "the man forces the young girl to the ground and begins to pummel her with a large stone.", "correct_sent": "the man forces the young girl to the ground and begins to pummel her.", "incorrect_sent": "the man forces the ground and begins to pummel her with a large stone."}
+{"article_sent": "google dubbed the first set of several thousand glass users.", "correct_sent": "google dubbed the first set of several thousand glass users.", "incorrect_sent": "google dubbed the first set of several thousand glass glass users."}
+{"article_sent": "professor jonathan armbruster, a biologist and curator of fishes at the museum, said his colleagues chris hamilton noticed the resemblance to the alien killed by han solo in the first of the original star wars films.", "correct_sent": "professor jonathan armbruster said his colleagues noticed resemblance to the alien killed by han solo.", "incorrect_sent": "the resemblance to the first of the original star wars films."}
+{"article_sent": "chelsea's izzy brown (left), has also been named in sean o'driscoll's side for the european qualifiers.", "correct_sent": "chelsea's izzy brown has been named in sean o'driscoll's european qualifiers.", "incorrect_sent": "chelsea has been named in sean o'driscoll's side for the european qualifiers."}
+{"article_sent": "the diminutive sub can dive to depths of up to 300ft (91 metres) so passengers can look at fish while staying dry and is small enough to be stowed on a yacht.", "correct_sent": "passengers can look at fish while staying dry and is small.", "incorrect_sent": "passengers can look at fish while staying on a yacht."}
+{"article_sent": "the ex-chief of defence staff said he was 'mystified' about why the police had searched his home in surrey.", "correct_sent": "the ex-chief of defence staff said he was 'mystified' about why the police had searched.", "incorrect_sent": "the ex-chief of defence staff said he was 'mystified' about the police."}
+{"article_sent": "now, though, kane is one of the most talked about players in the country as his roy of the rovers rise continues apace.", "correct_sent": "kane is one of the most talked about the players in the country.", "incorrect_sent": "kane is one of the most talked about players in the rovers."}
+{"article_sent": "and while the sport's ban in britain over animal welfare concerns 10 years ago is still debated, the american hunt is a chase - the intent is not to kill a fox.", "correct_sent": "the american hunt is a chase - the intent is not to kill.", "incorrect_sent": "the american hunt is a ban in britain over animal welfare concerns."}
+{"article_sent": "the victim, now 38, was one of two women attacked while under general anaesthetic by andrew hutchinson, 29, as they lay in hospital beds at john radcliffe hospital in oxford.", "correct_sent": "victim, now 38, is one of two women attacked while under general anaesthetic.", "incorrect_sent": "victim, 29, was one of two women attacked while under general anaesthetic."}
+{"article_sent": "(cnn) tina fey's follow-up to 30 rock is getting a lot of attention, not all of it good.", "correct_sent": "tina fey's follow-up to 30 rock is getting a lot of attention.", "incorrect_sent": "tina fey's follow-up to 30 rock is not all of it good."}
+{"article_sent": "'ninety-nine per cent of the things were very good,' said the world cup winner turned skills coach.", "correct_sent": "` ninety-nine per cent of things were very good,' said the world cup winner.", "incorrect_sent": "the world cup winner has been 'ninety-nine per cent of the things were very good'"}
+{"article_sent": "later wednesday, a top government spokesman dismissed as a hoax a report that singapore's founding father had died.", "correct_sent": "top government spokesman dismissed as a hoax report.", "incorrect_sent": "top government spokesman says singapore's founding father had died."}
+{"article_sent": "ireland will attempt to rack up both the win and tries needed at bt murrayfield on saturday afternoon to apply a significant dose of pressure to championship rivals england before they take on france in the evening kick-off.", "correct_sent": "ireland take on france in the evening kick-off on saturday.", "incorrect_sent": "ireland will take on france in the evening kick-off on sunday."}
+{"article_sent": "manchester united have also been to watch the argentine but he is not high on their list.", "correct_sent": "manchester united have also been to watch the argentine.", "incorrect_sent": "manchester united have also been to play the argentine."}
+{"article_sent": "lille's on-loan midfielder marcos lopes would be keen to stay with the ligue 1 club beyond the end of the season if he is not sure of regular playing time back at manchester city.", "correct_sent": "marcos lopes will be keen to stay with the ligue 1 club.", "incorrect_sent": "marcos rojo will be keen to stay with the ligue 1 club."}
+{"article_sent": "he has spoken of personally meeting prime minister david cameron but leads a double life as a seedy drug addict and regular user of prostitutes.", "correct_sent": "he has spoken of personally meeting prime minister but leads a double life.", "incorrect_sent": "david cameron leads a double life as a seedy drug addict."}
+{"article_sent": "samantha green (above in court), whose 20-day-old son was found dead near the sacramento river in northern california, has been charged with involuntary manslaughter and abusing or endangering the health of a child.", "correct_sent": "samantha green, whose 20-day-old son, was found dead near the sacramento river in northern california.", "incorrect_sent": "samantha green, 23, was found dead near the sacramento river in northern california."}
+{"article_sent": "moores, a county championship winner with both sussex and lancashire, finds himself under increasing pressure to prove he has what it takes to succeed at international level after england's woeful showing at this world cup.", "correct_sent": "moores, a county championship winner, finds himself under increasing pressure.", "incorrect_sent": "moores, a county championship, finds himself under increasing pressure."}
+{"article_sent": "got a spare $ 100 million? this is one of the numerous cozy living rooms in bill koch's private aspen retreat that is up for sale.", "correct_sent": "the numerous cozy living rooms in koch's private aspen retreat is up for sale.", "incorrect_sent": "the numerous cozy living rooms in bill koch's aspen retreat is up for sale."}
+{"article_sent": "the members of the latter-day saints church were visiting the branch in hamilton, the new zealand herald reported.", "correct_sent": "the members of the latter-day saints church were visiting the branch in hamilton.", "incorrect_sent": "the members of the latter-day saints church were visiting branch in hamilton."}
+{"article_sent": "the diminutive sub can dive to depths of up to 300ft (91 metres) so passengers can look at fish while staying dry and is small enough to be stowed on a yacht.", "correct_sent": "passengers can look at fish while staying dry and is small enough.", "incorrect_sent": "passengers can look at fish while stowed on a yacht."}
+{"article_sent": "speaking out: kai hibbard (seen left and right) has reportedly penned an expose on'the biggest loser.", "correct_sent": "hibbard has penned an expose on'the biggest loser.", "incorrect_sent": "kai hibbard has been penned a expose on'the biggest loser."}
+{"article_sent": "stuart pearce believes gary neville needs to get managerial experience of his own before he can be considered as a possible successor to roy hodgson as england head coach.", "correct_sent": "stuart pearce believes gary neville needs to get managerial experience.", "incorrect_sent": "stuart pearce believes gary neville needs to get managerial job."}
+{"article_sent": "giorgio armani gave us a lesson in luxury by creating what vogue are dubbing the 'skouser' - a wrap-around skirt-cum-pair of slim-fitting trousers made from one piece of material, which were paired with fur jackets and blouses.", "correct_sent": "giorgio armani gave us a lesson in luxury by creating what.", "incorrect_sent": "giorgio armani gave us a lesson in luxury by creating vogue."}
+{"article_sent": "but now the comedic duo have taken the unlikely collaboration off the runway in order to pose up for a quirky print campaign, which is believed to have been shot around the same time as the catwalk show.", "correct_sent": "the comedic duo have taken the unlikely collaboration off the runway.", "incorrect_sent": "the comedic duo is believed to have been shot around the same time."}
+{"article_sent": "the easter bunny was invented in german folklore, where an 'easter hare' was used to determine whether children had been well-behaved.", "correct_sent": "the easter bunny was invented in german folklore, where an 'easter hare.", "incorrect_sent": "the easter bunny was invented in german folklore, where children had been well-behaved."}
+{"article_sent": "locke chats with ryan stevenson last season.", "correct_sent": "locke chats with ryan stevenson.", "incorrect_sent": "locke chats with ryan stevenson last year."}
+{"article_sent": "his former teacher, 35-year-old brianne altice, has been charged with 14 felonies - including rape, sodomy and sex abuse - for allegedly having sexual relationships with three different students.", "correct_sent": "his former teacher, brianne altice, has been charged with rape, sodomy and sex abuse.", "incorrect_sent": "brianne altice, 21, has been charged with rape, sodomy and sex abuse."}
+{"article_sent": "the victim, now 38, was one of two women attacked while under general anaesthetic by andrew hutchinson, 29, as they lay in hospital beds at john radcliffe hospital in oxford.", "correct_sent": "victim, now 38, was one of two women attacked while under general anaesthetic.", "incorrect_sent": "victim, 29, was one of two women attacked while under general anaesthetic."}
+{"article_sent": "lane's husband found the infant in a bathtub and rushed the child to the hospital, where she was pronounced dead.", "correct_sent": "lane's husband found the infant in a bathtub and rushed the child to hospital.", "incorrect_sent": "lane's husband was found in a bathtub and rushed the child to the hospital."}
+{"article_sent": "they believe someone smashed their heads together as they stood outside the rose and crown on newgate street.", "correct_sent": "someone smashed their heads together as they stood outside the rose and crown on newgate street.", "incorrect_sent": "they believe someone smashed their heads together as they stood outside the rose."}
+{"article_sent": "it was all part of a charity concert put on by the rwanga foundation, a non-profit created to help those in the area, and in a press release the group claims that nelly is now 'the first american artist to perform in the city's 8,000 year old history.", "correct_sent": "nelly is the first american artist to perform in the city's 8,000 year old history.", "incorrect_sent": "the group is now the first american artist to perform in the city's 8,000 year old."}
+{"article_sent": "of that $ 90 million in expenditure, $ 45 million was used for entertainment items such as flat screen televisions, smart phones, sound systems and other home theatre devices.", "correct_sent": "$ 90 million in expenditure, $ 45 million was used for entertainment items.", "incorrect_sent": "$ 90 million in expenditure, smart phones, sound systems and other home theatre devices."}
+{"article_sent": "30 pm today were met with an error screen and an image of the old test card clown after the bbc encountered an internal system failure.", "correct_sent": "those trying to log on to the site were met with an error screen.", "incorrect_sent": "the old test card clown was met with an error screen."}
+{"article_sent": "australia all-rounder james faulkner won't be surprised if there's some niggle and verbal exchanges, just like he's not surprised that defending champion india shook off their poor form leading into the world cup to win seven consecutive matches.", "correct_sent": "australia all-rounder james faulkner won't be surprised if there's verbal exchanges.", "incorrect_sent": "australia all-rounder james faulkner won't be surprised by india's poor form."}
+{"article_sent": "but now conditions on the farms that supply kfc with 23million chickens a year can be shown for the first time.", "correct_sent": "but now conditions on the farms that supply kfc can be shown for the first time.", "incorrect_sent": "kfc with 23million chickens a year can be shown for the first time."}
+{"article_sent": "australia all-rounder james faulkner won't be surprised if there's some niggle and verbal exchanges, just like he's not surprised that defending champion india shook off their poor form leading into the world cup to win seven consecutive matches.", "correct_sent": "australia all-rounder james faulkner won't be surprised if there's some niggle.", "incorrect_sent": "australia all-rounder james faulkner won't be surprised by india's poor form."}
+{"article_sent": "my great-great-grandfather, jean bourdain, was one of them.", "correct_sent": "my great-great-grandfather, jean bourdain, was one of them.", "incorrect_sent": "my great-great-grandfather, jean bourdain, is one of them."}
+{"article_sent": "chelsea are maintaining a watching brief on anderlecht teenagers youri tielemans and leander dendoncker.", "correct_sent": "chelsea are maintaining a watching brief on anderlecht teenagers youri.", "incorrect_sent": "chelsea are maintaining a watching brief on anderlecht."}
+{"article_sent": "the emergencies ministry for moscow said the tower itself did not suffer significant damage and the fire's spread had been halted around midnight.", "correct_sent": "the fire's spread had been halted around midnight.", "incorrect_sent": "the tower itself did not have been halted around midnight."}
+{"article_sent": "fulham striker mccormack gets his shot away as fulham take on watford in the championship.", "correct_sent": "fulham striker mccormack gets his shot away as fulham take on watford.", "incorrect_sent": "fulham striker mccormack gets his shot away as fulham win on watford."}
+{"article_sent": "what's not to like about a ferry voyage to a fascinating outpost of our beautiful british isles? this has to be one of the most hassle-free ways to travel - where passengers can take their own car and the journey is all part of the experience.", "correct_sent": "passengers can take their car and the journey is all part of the experience.", "incorrect_sent": "passengers can take their own car and the journey is all part."}
+{"article_sent": "stuart pearce believes gary neville needs to get managerial experience of his own before he can be considered as a possible successor to roy hodgson as england head coach.", "correct_sent": "stuart pearce says neville needs to get managerial experience of his own.", "incorrect_sent": "stuart pearce believes gary neville needs to get managerial job."}
+{"article_sent": "insiders say the row brought simmering tensions between the starkly contrasting pair -- both rivals for miliband's ear -- to a head.", "correct_sent": "insiders say the row brought tensions between the starkly contrasting contrasting pair.", "incorrect_sent": "insiders say the row brought simmering tensions between miliband's ear."}
+{"article_sent": "with two successful sell-out edits already under her belt, the 27-year-old is once again collaborating with oasis to launch her loved by mollie edit - a carefully selected pick of her favourite pieces from the high-street store.", "correct_sent": "the 27-year-old is once again collaborating with oasis to launch her favourite pieces.", "incorrect_sent": "the 27-year-old is once again collaborating with oasis to launch her loved."}
+{"article_sent": "arsenal and manchester city are tracking bayer leverkusen midfielder hakan calhanoglu.", "correct_sent": "arsenal and manchester city also tracking bayer leverkusen midfielder hakan calhanoglu.", "incorrect_sent": "arsenal and hakan calhanoglu are tracking bayer leverkusen midfielder hakan calhanoglu."}
+{"article_sent": "lane's husband found the infant in a bathtub and rushed the child to the hospital, where she was pronounced dead.", "correct_sent": "lane's husband found infant in a bathtub and rushed the child to hospital.", "incorrect_sent": "lane's husband was found in a bathtub and rushed the child to the hospital."}
+{"article_sent": "it was all part of a charity concert put on by the rwanga foundation, a non-profit created to help those in the area, and in a press release the group claims that nelly is now 'the first american artist to perform in the city's 8,000 year old history.", "correct_sent": "the group is now the first american artist to perform in the city's old history.", "incorrect_sent": "the group is now the first american artist to perform in the city's 8,000 year old."}
+{"article_sent": "that is due to run out in early april but phelps' punishment also resulted in him being excluded from the us team for the world championships in kazan.", "correct_sent": "phelps is due to be excluded from the us team for the world championships.", "incorrect_sent": "phelps' punishment was due to run out in early hours."}
+{"article_sent": "seven games involving nimes were investigated after conrad was arrested last november.", "correct_sent": "seven games involving nimes were investigated after conrad's arrest.", "incorrect_sent": "seven games involving nimes were investigated after conrad was investigated."}
+{"article_sent": "they believe someone smashed their heads together as they stood outside the rose and crown on newgate street.", "correct_sent": "someone smashed their heads together as they stood outside the rose and crown on newgate street.", "incorrect_sent": "they believe someone smashed their heads together as they stood outside the crown."}
+{"article_sent": "bale has received a lot of criticism, but silenced his doubters with a la liga double at the bernabeu.", "correct_sent": "bale has received a lot of criticism but silenced his doubters.", "incorrect_sent": "bale has received a lot of criticism but silenced his doubters doubters."}
+{"article_sent": "visitors to the isle of lewis will be hoping the clouds stay away on friday for the solar eclipse.", "correct_sent": "visitors to the isle of lewis will be hoping the clouds stay away.", "incorrect_sent": "visitors to the isle of lewis will be hoping to stay away on friday."}
+{"article_sent": "but despite initially keeping it a secret from even those closest to him, he has now admitted -- behind closed doors -- to using prodigious quantities of the dangerous stimulant and of hiring prostitutes on a weekly basis.", "correct_sent": "but despite initially keeping it a secret from even those closest to him.", "incorrect_sent": "but he has now admitted to closed doors to using prodigious quantities of hiring prostitutes."}
+{"article_sent": "taylor swift has been a star since the age of 16, when she released her first album and it went to no.", "correct_sent": "taylor swift has been a star since she released her first album.", "incorrect_sent": "taylor swift has been a fan since the age of 16."}
+{"article_sent": "rima karaki was conducting an interview with london-based islamist hani al-seba' i for lebanon's al jadeed on the weekend of international women's day, discussing reports of christians joining islamic groups such as isis in the middle east when the scholar went off on a historical tangent.", "correct_sent": "rima karaki was conducting an interview with london-based islamist hani al-seba.", "incorrect_sent": "rima karaki was conducting an interview with london-based islamist al-seba al-seba."}
+{"article_sent": "pakistan fast bowler mohammad irfan will miss the remainder of the world cup with a pelvis injury.", "correct_sent": "mohammad irfan will miss the remainder of world cup with a pelvis injury.", "incorrect_sent": "mohammad irfan will miss the remainder of the world cup with a pelvis."}
+{"article_sent": "chelsea are maintaining a watching brief on anderlecht teenagers youri tielemans and leander dendoncker.", "correct_sent": "chelsea are maintaining a watching brief on anderlecht teenagers youri.", "incorrect_sent": "anderlecht are maintaining a watching brief on anderlecht."}
+{"article_sent": "the fish was discovered in the gurupi river in north central brazil and has large dark eyes, protruding birstles and a sucker mouth.", "correct_sent": "fish was discovered in the gurupi river in north central brazil.", "incorrect_sent": "the fish was discovered in the gurupi river in north brazil."}
+{"article_sent": "just in june, the administration declared the success of its deal to remove al-assad's chemical weapons, which had been prompted by his august 2013 sarin attacks on syrian civilians and the threat of u.", "correct_sent": "administration declared the success of its deal to remove al-assad's chemical weapons.", "incorrect_sent": "the president declared the success of its deal to remove al-assad's chemical weapons."}
+{"article_sent": "it's the latest attempt by the people's liberation army's air force to attract more talent.", "correct_sent": "the latest attempt by the people's air force to attract more talent.", "incorrect_sent": "the latest attempt by the people's air force will attract more talent."}
+{"article_sent": "today it's alzheimer's, and 55% of patients and their caregivers say their doctors never told them they have the devastating disease, according to a special report of the alzheimer's association released this week.", "correct_sent": "alzheimer's, 55% of patients, say their doctors never told them they have the devastating disease.", "incorrect_sent": "alzheimer's, 55% of patients and their caregivers say they have the devastating disease."}
+{"article_sent": "former manchester city boss roberto mancini has turned up the heat on successor manuel pellegrini by claiming the chilean should be winning a trophy every year.", "correct_sent": "roberto mancini has turned up the heat on manuel pellegrini's successor.", "incorrect_sent": "roberto mancini turned up the heat on manuel pellegrini's successor."}
+{"article_sent": "just in june, the administration declared the success of its deal to remove al-assad's chemical weapons, which had been prompted by his august 2013 sarin attacks on syrian civilians and the threat of u.", "correct_sent": "the administration declared the success of its deal to remove al-assad's chemical weapons.", "incorrect_sent": "the president declared the success of its deal to remove al-assad's chemical weapons."}
+{"article_sent": "foreigners from outside the eu will be forced to provide a criminal records certificate from their home country before being given a visa to enter the uk, the mail can reveal.", "correct_sent": "foreigners from outside the eu will be forced to provide a criminal records certificate.", "incorrect_sent": "eu will be forced to provide a criminal records certificate from their home country."}
+{"article_sent": "the 21-year-old is valued at # 30million by palermo president mauro zamparini and juventus are considered favourites even though they have had an opening offer rejected.", "correct_sent": "the 21-year-old is valued at # 30million by palermo president.", "incorrect_sent": "the 21-year-old is valued at # 30million for palermo president mauro zamparini."}
+{"article_sent": "badger was shot in the head at a constituent event for giffords, who was the target of gunman jared loughner.", "correct_sent": "badger was shot in the head at a constituent event for giffords.", "incorrect_sent": "giffords was shot in the head at a constituent event for giffords."}
+{"article_sent": "the lion died while it was being transported.", "correct_sent": "the lion died while it was being being transported.", "incorrect_sent": "the lion died while it was being transported to hospital."}
+{"article_sent": "but now the comedic duo have taken the unlikely collaboration off the runway in order to pose up for a quirky print campaign, which is believed to have been shot around the same time as the catwalk show.", "correct_sent": "the comedic duo have taken the unlikely collaboration off the runway.", "incorrect_sent": "the comedic duo have been shot around the same time."}
+{"article_sent": "that is due to run out in early april but phelps' punishment also resulted in him being excluded from the us team for the world championships in kazan.", "correct_sent": "phelps is due to be excluded from the us team for the world championships.", "incorrect_sent": "phelps' punishment was due to run out for the world championships in kazan."}
+{"article_sent": "a mother-of-three says she was shocked to open a bag of her favourite crisps and discovered the world's smallest wotsits inside.", "correct_sent": "she was shocked to open a bag of her favourite crisps.", "incorrect_sent": "she was shocked to open a bag of her favourite food."}
+{"article_sent": "kai hibbard worked on her book 'too fat, too thin, can't win' with deborah baer, according to the new york daily news.", "correct_sent": "kai hibbard worked on her book 'too fat, too thin' with deborah baer.", "incorrect_sent": "kai hibbard worked on her book 'too fat' with deborah baer."}
+{"article_sent": "in the loungeroom, he hid about 50 pairs of shoes under the table and couch and hid hundreds more in the kitchen and bathroom.", "correct_sent": "in the loungeroom, he hid about 50 pairs of shoes under the table and couch.", "incorrect_sent": "he hid about 50 pairs of shoes under the table and hid hundreds more."}
+{"article_sent": "plagad somian, 23, began hassling marek pietrzycka, 45, when she saw him walking with his wife katarzyna in the street in pattaya, a city on the east coast of thailand known as a sex tourist hotspot.", "correct_sent": "plagad somian, 23, saw him walking with his wife katarzyna in the street.", "incorrect_sent": "plagad somian, 23, began walking with his wife katarzyna in the street in pattaya."}
+{"article_sent": "later wednesday, a top government spokesman dismissed as a hoax a report that singapore's founding father had died.", "correct_sent": "top government spokesman dismissed as a hoax report.", "incorrect_sent": "top government says singapore's founding father had died."}
+{"article_sent": "families reacted with fury yesterday at learning that tony blair will attend a service honouring soldiers killed in afghanistan.", "correct_sent": "families reacted with fury yesterday at learning that tony blair will attend a service.", "incorrect_sent": "families reacted with fury yesterday at learning that tony blair killed in afghanistan."}
+{"article_sent": "after four straight six nations defeats, it may seem that he is a little detached from the reality of the situation but jim hamilton, that most rugged of lock forwards, maintains this is a team on the verge of something special.", "correct_sent": "jim hamilton is a little detached from the reality of the world.", "incorrect_sent": "jim hamilton is a little detached from the reality of something special."}
+{"article_sent": "not only that, but his long term partner once declared herself bankrupt over an unpaid # 120,000 tax bill -- despite freeman having an estimated fortune of # 10million.", "correct_sent": "but his long term partner once declared herself bankrupt over an unpaid tax bill.", "incorrect_sent": "but he declared herself bankrupt over an unpaid # 120,000 tax bill."}
+{"article_sent": "the amazing collection of clips include some that have been artificially amplified to be audible to humans, and others that are simply as they originally sounded.", "correct_sent": "the amazing collection of clips include some that have been artificially amplified to humans.", "incorrect_sent": "the amazing collection of clips include some that have been artificially audible to humans."}
+{"article_sent": "samantha green (above in court), whose 20-day-old son was found dead near the sacramento river in northern california, has been charged with involuntary manslaughter and abusing or endangering the health of a child.", "correct_sent": "samantha green, whose 20-day-old son, was found dead near the sacramento river in northern california.", "incorrect_sent": "samantha green was found dead near the sacramento river in northern california."}
+{"article_sent": "manchester united have also been to watch the argentine but he is not high on their list.", "correct_sent": "manchester united have also been linked to the argentine.", "incorrect_sent": "manchester united have also been to watch the world cup."}
+{"article_sent": "a former member of the islamic state today claimed he witnessed the british graduate dubbed jihadi john behead a japanese hostage.", "correct_sent": "he claimed he witnessed the british graduate dubbed jihadi john behead a japanese hostage.", "incorrect_sent": "former member of the islamic state today claimed he was dubbed jihadi john behead a japanese hostage."}
+{"article_sent": "insiders say the row brought simmering tensions between the starkly contrasting pair -- both rivals for miliband's ear -- to a head.", "correct_sent": "insiders say the row brought tensions between the starkly contrasting contrasting pair.", "incorrect_sent": "insiders say row brought tensions between miliband's ear and head."}
+{"article_sent": "foreigners from outside the eu will be forced to provide a criminal records certificate from their home country before being given a visa to enter the uk, the mail can reveal.", "correct_sent": "foreigners from outside eu will be forced to provide a criminal records certificate.", "incorrect_sent": "eu will be forced to provide a criminal records certificate from their home country."}
+{"article_sent": "it said the second draft law, which is to go before parliament next month, would close many jobs to women who choose not to or are unable to have children.", "correct_sent": "second draft law is to go before parliament next month.", "incorrect_sent": "second draft law would close many jobs to women who have children."}
+{"article_sent": "in stuart lancaster's fourth year in charge they are yet to win this competition.", "correct_sent": "stuart lancaster's fourth year in charge are yet to win this competition.", "incorrect_sent": "stuart lancaster's fourth year in charge they will win this competition."}
+{"article_sent": "a tweet from sportscaster fred katz challenged the today show's willie geist and uzo aduba, who plays suzanne crazy eyes warren on orange is the new black, to share a lady and the tramp licorice.", "correct_sent": "crazy eyes warren on orange is the new black", "incorrect_sent": "sportscaster fred katz's crazy eyes warren on orange is the new black"}
+{"article_sent": "manchester united have also been to watch the argentine but he is not high on their list.", "correct_sent": "manchester united have also been to watch the argentine.", "incorrect_sent": "manchester united have also been to watch the world cup."}
+{"article_sent": "giorgio armani gave us a lesson in luxury by creating what vogue are dubbing the 'skouser' - a wrap-around skirt-cum-pair of slim-fitting trousers made from one piece of material, which were paired with fur jackets and blouses.", "correct_sent": "giorgio armani gave us a lesson in luxury by creating what vogue.", "incorrect_sent": "giorgio armani gave us a lesson in luxury by creating vogue."}
+{"article_sent": "to do this, the panther chameleon shoots out its tongue to almost twice its body length in little more than 0.", "correct_sent": "panther chameleon shoots out its tongue to almost twice its body length in little more than 0.07 seconds.", "incorrect_sent": "the panther shoots out its tongue to almost twice its body length in little more than 0.07 seconds."}
+{"article_sent": "now, a device called the crow box could enable bird watchers to make money from their hobby, as well as watch birds develop new skills.", "correct_sent": "a device called the crow box could enable bird watchers to make money.", "incorrect_sent": "a device called the crow box could make money from their hobby."}
+{"article_sent": "the 45-year-old replaced alex mcleish at the helm of the premier league club in june 2012 but following a poor run of results, which left villa in the bottom three, the club parted company with the scot and later appointed tim sherwood as his successor.", "correct_sent": "the 45-year-old replaced alex mcleish at the helm of the premier league.", "incorrect_sent": "the 45-year-old replaces alex mcleish at the helm of the premier league."}
+{"article_sent": "undeterred by the fact that he was unable to use his hands and lying next to several police officers, the man struggles to his feet while they are tackling two people nearby and sprints off through a watching crowd.", "correct_sent": "he was unable to use his hands and sprints off through a watching crowd.", "incorrect_sent": "the man struggles to use his hands and lying next to police officers."}
+{"article_sent": "the gliese 581d planet has conditions that could support life, and is likely to be a rocky world, twice the size of earth.", "correct_sent": "the gliese 581d 581d planet has conditions that could support life.", "incorrect_sent": "the gliese 581d planet has conditions twice the size of earth."}
+{"article_sent": "arsenal and manchester city are tracking bayer leverkusen midfielder hakan calhanoglu.", "correct_sent": "arsenal and manchester city are tracking bayer leverkusen midfielder hakan calhanoglu.", "incorrect_sent": "arsenal and hakan calhanoglu are tracking bayer leverkusen midfielder hakan calhanoglu."}
+{"article_sent": "not forgetting time spent enjoying the views and fresh air on the open deck, before it's time to disembark for an exciting island adventure.", "correct_sent": "not forgetting time spent enjoying the views and fresh air.", "incorrect_sent": "it's time to disembark for an exciting island adventure."}
+{"article_sent": "hussein javadi took the picture in the window of hotel hespiria before he departed for the airport on tuesday morning, sharing it on his private instagram account.", "correct_sent": "hussein javadi took the picture in the window of hotel hespiria.", "incorrect_sent": "hussein javadi took the picture in the window of hespiria hespiria."}
+{"article_sent": "the search for noah thomas, from dublin, has widened since he was first went missing when his mother decided to get another hour of sleep on sunday morning.", "correct_sent": "the search for noah thomas has widened since he was first went missing.", "incorrect_sent": "noah thomas, from dublin, has widened since he was first missing."}
+{"article_sent": "but six months later, the three-time olympic gold medalist's office at the misty may-treanor sports center is empty after a mysterious fallout with her longtime trainer mike rangel, who founded the irvine facility.", "correct_sent": "the three-time olympic gold medalist's office is empty after a mysterious fallout.", "incorrect_sent": "but six months later, the three-time olympic gold medalist is empty."}
+{"article_sent": "we reported that, just before we battened down the hatches, all the three and four-year-old boys had been snapping at their pyjama elastic and comparing the size of their willies.", "correct_sent": "the three and four-year-old boys were snapping at their pyjama elastic.", "incorrect_sent": "the three and four-year-old boys had been pictured snapping at their pyjama elastic."}
+{"article_sent": "the mother-of-one added that the problem of young people not understanding the realities of sex is at least as bad now as it has ever been.", "correct_sent": "the mother-of-one said the problem of young people not understanding the realities of sex is at least.", "incorrect_sent": "the mother-of-one said the problem of sex is at least as bad now as it has ever been."}
+{"article_sent": "however, there is also interest from roma while tottenham and everton have asked to be kept informed.", "correct_sent": "tottenham and everton have asked to be kept informed.", "incorrect_sent": "roma and everton have asked to be kept informed."}
+{"article_sent": "what we already know about the foundation's donors is controversial: some of the corporations, banks and foreign governments that have funded foundation projects have abysmal human rights records and have been involved in everything from drug money laundering to aiding terrorism in violation of american law.", "correct_sent": "the foundation's donors have been involved in everything from drug money laundering.", "incorrect_sent": "the foundation's donors have been involved in everything from drug money laundering."}
+{"article_sent": "the mother-of-one added that the problem of young people not understanding the realities of sex is at least as bad now as it has ever been.", "correct_sent": "the mother-of-one said the realities of sex is at least as bad now as it has ever been.", "incorrect_sent": "the mother-of-one said the problem of sex is at least as bad now as it has ever been."}
+{"article_sent": "a man arrested outside a bar in austin, texas, during the sxsw (south by southwest) festival was filmed making a brazen bid for freedom -- after being handcuffed and made to lie on the floor.", "correct_sent": "a man arrested outside a bar in austin, texas, after being handcuffed.", "incorrect_sent": "a man arrested outside a bar in austin, texas, after being arrested."}
+{"article_sent": "the bolton boxer had hoped to take on floyd mayweather but the pound-for-pound king will instead meet manny pacquaio in a $ 300million mega-fight on may 2.", "correct_sent": "the bolton boxer will instead meet manny pacquaio in a $ 300million mega-fight.", "incorrect_sent": "the bolton boxer had hoped to take the floyd mayweather in a $ 300million mega-fight."}
+{"article_sent": "the bolton boxer had hoped to take on floyd mayweather but the pound-for-pound king will instead meet manny pacquaio in a $ 300million mega-fight on may 2.", "correct_sent": "the bolton boxer had hoped to take on floyd mayweather.", "incorrect_sent": "the bolton boxer had hoped to take the floyd mayweather in a $ 300million mega-fight."}
+{"article_sent": "we are on the very spot where matthew crawley asked lady mary to marry him in the first downton abbey christmas special, broadcast in 2011.", "correct_sent": "matthew crawley asked lady mary to marry him in first downton abbey christmas special.", "incorrect_sent": "matthew crawley asked lady mary to marry him in the first downton abbey."}
+{"article_sent": "valentin pavlov claimed he was teaching eugene efimov how to kick the ball properly after the lad failed to show enough commitment in a tackle.", "correct_sent": "valentin pavlov claims he was teaching eugene efimov how to kick the ball properly.", "incorrect_sent": "valentin pavlov claimed he was teaching eugene efimov how to show enough commitment."}
+{"article_sent": "architect andrew kinslow told daily mail online that the plan 'has kind of taken on a life of its own' after it was released late last month.", "correct_sent": "architect kinslow kinslow said the plan 'has kind of its own'", "incorrect_sent": "architect andrew kinslow said the plan 'has kind of its own'"}
+{"article_sent": "insiders say the row brought simmering tensions between the starkly contrasting pair -- both rivals for miliband's ear -- to a head.", "correct_sent": "insiders say the row brought tensions between the contrasting pair.", "incorrect_sent": "insiders say row brought tensions between miliband's ear and head."}
+{"article_sent": "compare that to one of the big four cancers -- breast, colorectal, lung and prostate -- more than 90% said their doctors had no problem giving them the diagnosis.", "correct_sent": "more than 90% said their doctors had no problem giving them the diagnosis.", "incorrect_sent": "more than 90% said they had no problem giving them the diagnosis."}
+{"article_sent": "that is due to run out in early april but phelps' punishment also resulted in him being excluded from the us team for the world championships in kazan.", "correct_sent": "phelps' punishment was excluded from the us team for the world championships.", "incorrect_sent": "phelps' punishment was due to run out in early hours."}
+{"article_sent": "a blind man has been investigated for benefit fraud after he scored a 180 in a charity darts event and was deemed to be too good to be blind.", "correct_sent": "he scored a 180 in a charity darts event.", "incorrect_sent": "he scored a 180 charity darts event and was deemed to be blind."}
+{"article_sent": "both on screen and on the red carpet, she always radiates with a natural beauty - but while cate has avoided going under the knife or being jabbed with needles, she still relies on some beauty secrets to keep her looking her best.", "correct_sent": "she always radiates with a natural beauty and relies on some beauty secrets.", "incorrect_sent": "she always radiates with needles, but cate has avoided going under the knife."}
+{"article_sent": "(cnn) as the world awaits a possible nuclear agreement between the united states and iran, let us not forget: this is the second time in two years that president barack obama has pursued a nonproliferation deal with an unreformed regime in the middle east.", "correct_sent": "obama has pursued a nonproliferation deal with an unreformed regime.", "incorrect_sent": "the world is the second time in two years."}
+{"article_sent": "it said the second draft law, which is to go before parliament next month, would close many jobs to women who choose not to or are unable to have children.", "correct_sent": "second draft law is to go before parliament next month.", "incorrect_sent": "second draft law would close many women who choose not to have children."}
+{"article_sent": "(cnn) hillary clinton's lawyerly news conference tuesday, at which she doggedly parried suggestions that her use of a private email server to conduct official state department business was improper, might extinguish the matter for most americans -- but it doesn't get clinton out of the woods politically.", "correct_sent": "hillary clinton's lawyerly parried suggestions that her use of a private email server might extinguish the matter.", "incorrect_sent": "hillary doggedly suggestions that her use of a private email server might extinguish the matter for most americans."}
+{"article_sent": "bale has received a lot of criticism, but silenced his doubters with a la liga double at the bernabeu.", "correct_sent": "bale has received a lot of criticism but silenced his doubters.", "incorrect_sent": "bale has been silenced by a la liga double at the bernabeu."}
+{"article_sent": "when avery wallace was born, doctors told his parents: 'it's a girl'.", "correct_sent": "doctors told his mother: 'it's a girl'", "incorrect_sent": "doctors told his parents: 'it's a."}
+{"article_sent": "6 acres, with multiple cabins overlooking 'trout-soaked ponds' the four main cabins were originally built to be used as a bed and breakfast or wedding party venue but bill koch snapped it up.", "correct_sent": "the property is nicknamed the elk mountain lodge, with multiple cabins overlooking 'trout-soaked ponds'", "incorrect_sent": "the 28-bedroom property is nicknamed the mountain lodge, with multiple cabins overlooking 'trout-soaked ponds'"}
+{"article_sent": "an unconscious patient secretly raped in a&e by a male nurse who filmed the attack admitted today she tried to kill herself after police revealed her ordeal.", "correct_sent": "she was secretly raped in a&e by a male nurse who filmed the attack.", "incorrect_sent": "police say she tried to kill herself after police revealed her ordeal."}
+{"article_sent": "a user of a guide dog himself, he knew how important they could be for individuals with impaired sight.", "correct_sent": "a user of a guide dog knew how important they could be for individuals with impaired sight.", "incorrect_sent": "a user of a guide, he knew how important they could be for individuals with impaired sight."}
+{"article_sent": "but critics say they do not go far enough and will allow lenders to shut hundreds of branches across the country -- even when there is no other bank in the community.", "correct_sent": "critics say they do not go far enough and will allow lenders to shut up.", "incorrect_sent": "critics say they do not go far enough to shut hundreds of branches."}
+{"article_sent": "most players have been criticised - cristiano ronaldo included - but perhaps most under-fire is iker casillas.", "correct_sent": "cristiano ronaldo included cristiano ronaldo and iker casillas.", "incorrect_sent": "cristiano ronaldo included iker casillas and iker casillas."}
+{"article_sent": "of that $ 90 million in expenditure, $ 45 million was used for entertainment items such as flat screen televisions, smart phones, sound systems and other home theatre devices.", "correct_sent": "$ 45 million was used for entertainment items such as flat screen televisions, sound systems and other home theatre devices.", "incorrect_sent": "$ 90 million in expenditure, smart phones, sound systems and other home theatre devices."}
+{"article_sent": "the ex-chief of defence staff said he was 'mystified' about why the police had searched his home in surrey.", "correct_sent": "the ex-chief of defence staff said he was 'mystified' about why the police had searched his.", "incorrect_sent": "the ex-chief of defence staff said he was 'mystified' about the police."}
+{"article_sent": "jenson button was happy to see mclaren be more competitive at the malaysian grand prix.", "correct_sent": "jenson button was happy to see mclaren be competitive at the malaysian grand prix.", "incorrect_sent": "jenson button was happy to be more competitive at the malaysian grand prix."}
+{"article_sent": "the prince of wales described becoming a first-time grandfather as 'a unique moment in anyone's life', pictured at george's christening in 2013.", "correct_sent": "prince of wales described becoming a first-time grandfather as 'a unique moment'", "incorrect_sent": "prince of wales described becoming a first-time 'unique moment in anyone's life'"}
+{"article_sent": "the lion died while it was being transported.", "correct_sent": "the lion died while it was being transported.", "incorrect_sent": "the lion died while it was being transported to hospital."}
+{"article_sent": "ms bensley was found dead in a pool of blood with her head in the gutter in smethwick, west midlands, moments after the violent battering in september last year.", "correct_sent": "ms bensley was found dead in a pool of blood with her head in smethwick, west midlands.", "incorrect_sent": "ms bensley was found dead in a pool of her head in the gutter in smethwick, west midlands."}
+{"article_sent": "the current generation of young people is the most confused about sex ever because of pornography, according to a psychotherapist.", "correct_sent": "the current generation of young people is confused about sex ever because of pornography.", "incorrect_sent": "the current generation of young people is the most confused about pornography."}
+{"article_sent": "she and her husband were trying to walk away.", "correct_sent": "she and her husband were trying to walk away.", "incorrect_sent": "she and her wife were trying to walk away."}
+{"article_sent": "ex-nottingham forest boss pearce, who was in charge of england under-21s for six years and also took caretaker charge of the senior team following fabio capello's resignation, told bbc radio five live's sportsweek: 'you only really know what it's like to manage when you're in that technical area and you're the manager.", "correct_sent": "pearce has been in charge of england under-21s for six years.", "incorrect_sent": "neville has been in charge of england under-21s for six years."}
+{"article_sent": "one clip uses audio data collected by the huygens probe as it descended onto the surface of saturn's moon titan on 14 january 2005.", "correct_sent": "one clip uses audio data collected by the huygens probe.", "incorrect_sent": "one video uses audio data collected by the huygens probe."}
+{"article_sent": "five ambitious clubs are locked in a scramble for two champions league places behind chelsea and manchester city and it could prove more tense and exciting than the barclays premier league title race itself.", "correct_sent": "five clubs are locked in a scramble for two champions league places.", "incorrect_sent": "five ambitious clubs are locked in a bid for two champions league places."}
+{"article_sent": "france's ruling socialists suffered a crushing defeat in regional elections last night - as three years of left-wing rule was branded a 'massive failure'.", "correct_sent": "france's ruling socialists suffered crushing defeat in regional elections last night.", "incorrect_sent": "france's ruling department has been branded a 'massive failure'"}
+{"article_sent": "he was also ordered to pay $ 167 million in restitution and fines.", "correct_sent": "he was ordered to pay $ 167 million in restitution and fines.", "incorrect_sent": "he was also ordered to pay $ 167 million in damages and fines."}
+{"article_sent": "how many other people can say that? by my reckoning, it's no more than a handful: paul mccartney, mick jagger, bob dylan, clint eastwood and hm the queen.", "correct_sent": "paul mccartney, bob dylan, clint eastwood and hm the queen.", "incorrect_sent": "paul mccartney, bob dylan, clint eastwood and clint eastwood were the queen."}
+{"article_sent": "the gunners have been on the 21-year-old's case since last season, but city are also paying close attention to his displays having impressed etihad stadium scouts in recent months.", "correct_sent": "the gunners have been on the 21-year-old's case since last year.", "incorrect_sent": "the arsenal have been on the 21-year-old's case since last season."}
+{"article_sent": "katie launched an attack on danny's partner joanne mas, saying: 'no woman should ever propose to a man,' to which the eastenders star responded:' i agree.", "correct_sent": "katie launched an attack saying: 'no woman should ever propose to a man'", "incorrect_sent": "katie launched an attack on danny mas, saying: 'no woman should ever propose to a man'"}
+{"article_sent": "that is due to run out in early april but phelps' punishment also resulted in him being excluded from the us team for the world championships in kazan.", "correct_sent": "phelps' punishment was excluded from the us team for the world championships.", "incorrect_sent": "phelps' punishment was due to run out for the world championships in kazan."}
+{"article_sent": "ancelotti praised his side for their performance before the break in sunday's 2-0 win at home to levante but said he was concerned by the way his players relaxed and let the visitors off the hook in the second period.", "correct_sent": "ancelotti's side were at home to levante in the second period.", "incorrect_sent": "ancelotti's side were beaten by levante in their 2-0 win over levante."}
+{"article_sent": "i'm in a basement in central london, moments from the frenzy of oxford street.", "correct_sent": "i'm is in a basement in central london.", "incorrect_sent": "i'm was in a basement in central london."}
+{"article_sent": "adding to the sense of theatre is the fact that those who do not join europe's elite next season must swallow an estimated # 30million in lost revenue and the prospect of a campaign in the largely unloved europa league.", "correct_sent": "the europa league must swallow an estimated # 30million in lost revenue.", "incorrect_sent": "the europa league must swallow an estimated # 30m in lost revenue."}
+{"article_sent": "giorgio armani gave us a lesson in luxury by creating what vogue are dubbing the 'skouser' - a wrap-around skirt-cum-pair of slim-fitting trousers made from one piece of material, which were paired with fur jackets and blouses.", "correct_sent": "giorgio armani gave us a lesson in luxury by creating what.", "incorrect_sent": "giorgio armani gave the 'skouser' of slim-fitting trousers."}
+{"article_sent": "he told eva merkacheva and andrey babushkin he had been tortured and the well-being of his family and friends were threatened.", "correct_sent": "he told eva merkacheva and andrey babushkin he had been tortured and the well-being of his family.", "incorrect_sent": "eva merkacheva and andrey babushkin were tortured and the well-being of his family and friends were threatened."}
+{"article_sent": "the man, named only as saleh, said the masked militant who appears in several beheading videos was a senior figure in the extremist organisation responsible for murdering foreign captives.", "correct_sent": "the man, named as saleh, said the masked militant who appears in several beheading videos.", "incorrect_sent": "the man, named only as saleh, appears in several beheading videos."}
+{"article_sent": "mr salamone slept on the sofa when they got home and later went upstairs to find her dead, the inquest heard.", "correct_sent": "mr salamone slept on the sofa when they got home and later went upstairs.", "incorrect_sent": "mr salamone slept on the sofa when they went upstairs to find her dead."}
+{"article_sent": "a tweet from sportscaster fred katz challenged the today show's willie geist and uzo aduba, who plays suzanne crazy eyes warren on orange is the new black, to share a lady and the tramp licorice.", "correct_sent": "sportscaster fred katz challenged the crazy eyes warren on orange is the black", "incorrect_sent": "sportscaster fred katz's crazy eyes warren on orange is the new black"}
+{"article_sent": "a jury at wolverhampton crown court heard how the duo had planned to carry out a robbery via text messages a fortnight before the murder.", "correct_sent": "jury heard the duo had planned to carry out a robbery via text messages.", "incorrect_sent": "jury told the duo had planned to carry out a robbery via text messages."}
+{"article_sent": "we are on the very spot where matthew crawley asked lady mary to marry him in the first downton abbey christmas special, broadcast in 2011.", "correct_sent": "matthew crawley asked lady mary to marry him in the first downton abbey christmas.", "incorrect_sent": "matthew crawley asked lady mary to marry him in the first downton abbey."}
+{"article_sent": "but nearly two years after this very public display of affection, charles is said to have become increasingly perplexed about the peripheral role he occupies in the young prince's life.", "correct_sent": "charles is said to have become increasingly perplexed over the peripheral role.", "incorrect_sent": "charles is said to have become increasingly increasingly perplexed about the peripheral role."}
+{"article_sent": "police have launched a probe into the 'unexplained death' of sophia walker (pictured) after she was found unresponsive in the early hours of saturday.", "correct_sent": "police have launched a investigation into the 'unexplained death' of sophia walker.", "incorrect_sent": "police have launched a probe into the 'tragic death' of sophia walker."}
+{"article_sent": "what's not to like about a ferry voyage to a fascinating outpost of our beautiful british isles? this has to be one of the most hassle-free ways to travel - where passengers can take their own car and the journey is all part of the experience.", "correct_sent": "passengers can take their own car and journey is all part of the experience.", "incorrect_sent": "passengers can take their own car and the journey is all part."}
+{"article_sent": "one clip uses audio data collected by the huygens probe as it descended onto the surface of saturn's moon titan on 14 january 2005.", "correct_sent": "the huygens probe descended onto the surface of saturn's moon titan on 14 january 2005.", "incorrect_sent": "one video uses audio data collected by the huygens probe."}
+{"article_sent": "(cnn) hillary clinton's lawyerly news conference tuesday, at which she doggedly parried suggestions that her use of a private email server to conduct official state department business was improper, might extinguish the matter for most americans -- but it doesn't get clinton out of the woods politically.", "correct_sent": "hillary clinton's lawyerly parried suggestions that her use of a private email server might extinguish the matter.", "incorrect_sent": "hillary clinton says her use of a private email server might extinguish the matter for most americans."}
+{"article_sent": "ms bensley was found dead in a pool of blood with her head in the gutter in smethwick, west midlands, moments after the violent battering in september last year.", "correct_sent": "ms bensley was found dead in a pool of blood with her head in the gutter in smethwick.", "incorrect_sent": "ms bensley was found dead in a pool of her head in the gutter in smethwick, west midlands."}
+{"article_sent": "they have been on death row since 2005 and have now been transferred to nusakambangan island where they are expected to be executed this week.", "correct_sent": "they have been transferred to nusakambangan island where they are expected to be executed.", "incorrect_sent": "they have been transferred to nusakambangan island where they are expected to be released."}
+{"article_sent": "beleaguered travellers have been warned that 'easter is cancelled' as far as trains are concerned and many key services will be axed.", "correct_sent": "beleaguered travellers warned that 'easter is cancelled' as far as trains.", "incorrect_sent": "beleaguered trains are concerned and many key services will be axed."}
+{"article_sent": "the family of condemned australian drug smuggler andrew chan, his mother helen (centre) and brother michael (wearing pink shorts) arrive at yogyakarta airport on thursday to make the five-hour drive to cilacap in central java and take a boat to nusakambangan island where chan is waiting to be executed.", "correct_sent": "family of condemned drug smuggler andrew chan, his mother helen, and brother michael.", "incorrect_sent": "family of condemned australian drug smuggler andrew chan, his mother helen, and brother helen."}
+{"article_sent": "left-armer irfan, who has taken eight wickets in five matches so far, underwent an mri scan after sitting out pakistan's seven-wicket win over ireland on sunday with pain in his hip.", "correct_sent": "left-armer irfan has taken eight wickets in five matches so far.", "incorrect_sent": "left-armer irfan has taken three wickets in five matches so far."}
+{"article_sent": "(cnn) as the world awaits a possible nuclear agreement between the united states and iran, let us not forget: this is the second time in two years that president barack obama has pursued a nonproliferation deal with an unreformed regime in the middle east.", "correct_sent": "obama has pursued a nonproliferation deal with an unreformed regime.", "incorrect_sent": "the world's first nuclear deal between the united states and iran."}
+{"article_sent": "chelsea are maintaining a watching brief on anderlecht teenagers youri tielemans and leander dendoncker.", "correct_sent": "chelsea are maintaining a watching brief brief on anderlecht.", "incorrect_sent": "anderlecht are maintaining a watching brief on anderlecht."}
+{"article_sent": "the 27 year-old serb's victory not only put him into the final of the bnp paribas open at indian wells, it also represented his ninth victory out of 10 against murray, the exception being the 2013 wimbledon final.", "correct_sent": "the 27 year-old serb's victory not only put him into the final.", "incorrect_sent": "the 27 year-old serb's victory not only put him into the final of 10."}
+{"article_sent": "the model had to have a combination of pmma and hydrogel removed from her body.", "correct_sent": "the model had to have a combination of pmma and hydrogel removed.", "incorrect_sent": "the model had to have a combination of pmma and hydrogel."}
+{"article_sent": "badger was shot in the head at a constituent event for giffords, who was the target of gunman jared loughner.", "correct_sent": "badger was shot in the head at a constituent conference for giffords.", "incorrect_sent": "giffords was shot in the head at a constituent event for giffords."}
+{"article_sent": "jenson button was happy to see mclaren be more competitive at the malaysian grand prix.", "correct_sent": "jenson button was happy to see mclaren be more competitive at malaysian grand prix.", "incorrect_sent": "jenson button was happy to be more competitive at the malaysian grand prix."}
+{"article_sent": "england coach mike catt wants england to be more ruthless when it comes to scoring points.", "correct_sent": "mike catt wants england to be more ruthless.", "incorrect_sent": "mike catt wants england to be more ruthless when it comes."}
+{"article_sent": "khalsa secondary academy, where kirsty jarvis does not want her 11-year-old daughter freya to go as she would be a 'minority' the sikh faith school opened in the village of stoke poges, buckinghamshire, last september despite protests from residents.", "correct_sent": "kirsty jarvis does not want her 11-year-old daughter freya to go as a 'minority'", "incorrect_sent": "khalsa secondary academy does not want her 11-year-old daughter freya to go."}
+{"article_sent": "the championship promotion hopefuls had made enquiries about signing jordan rhodes from blackburn rovers but the ewood park club's reluctance to deal has prompted fresh questions about mccormack.", "correct_sent": "the championship promotion hopefuls had made enquiries about signing jordan rhodes.", "incorrect_sent": "jordan rhodes has made enquiries from blackburn rovers."}
+{"article_sent": "the transgender 15-year-old knew he was a boy from the age of two, setting his physical 'parts' aside.", "correct_sent": "the 15-year-old knew he was a boy from the age of two.", "incorrect_sent": "the transgender 15-year-old knew he was a child from the age of two."}
+{"article_sent": "lloris wearing the scarf after defeat by chelsea in the capital one cup final at wembley.", "correct_sent": "lloris wearing scarf after defeat by chelsea in capital one cup final.", "incorrect_sent": "lloris wearing the scarf after defeat by chelsea in the fa cup final."}
+{"article_sent": "americans are about 5% of the world's population, yet we take half its pills, and 80% of its painkillers.", "correct_sent": "americans about 5% of the world's population, yet we take half its pills.", "incorrect_sent": "americans are about 5% of the world's population, yet we take its pills."}
+{"article_sent": "surrounded by her father, curt knox, mother, edda mellas and mr sutherland, knox, 27, broke down in tears as she addressed the press following italy's supreme court's ruling on friday night.", "correct_sent": "knox, 27, broke down in tears as she addressed the press.", "incorrect_sent": "knox, 27, broke down in tears after italy's supreme court's ruling."}
+{"article_sent": "he has spoken of personally meeting prime minister david cameron but leads a double life as a seedy drug addict and regular user of prostitutes.", "correct_sent": "he has spoken of personally meeting prime minister david cameron.", "incorrect_sent": "david cameron leads a double life as a seedy drug addict."}
+{"article_sent": "police have been forced to play prince charming after numerous residents complained that their shoes had gone missing from their front gardens.", "correct_sent": "police have been forced to play prince charming after numerous residents complained.", "incorrect_sent": "police have complained that their shoes had gone missing from their front gardens."}
+{"article_sent": "and a prized silver dollar each could fetch $ 10 million or more.", "correct_sent": "a prized silver dollar could fetch $ 10 million or more.", "incorrect_sent": "a prized silver dollar each could fetch $ 10 or more."}
+{"article_sent": "he claims he knew he was a boy when he was two.", "correct_sent": "he claims he was a boy when he was two.", "incorrect_sent": "he claims he was a young boy when he was two."}
+{"article_sent": "they believe someone smashed their heads together as they stood outside the rose and crown on newgate street.", "correct_sent": "they believe someone smashed their heads together as they stood outside the rose and crown.", "incorrect_sent": "they believe someone smashed their heads together as they stood outside the rose."}
+{"article_sent": "and while the sport's ban in britain over animal welfare concerns 10 years ago is still debated, the american hunt is a chase - the intent is not to kill a fox.", "correct_sent": "the american hunt is a chase - the intent is not to kill.", "incorrect_sent": "the american hunt is a chase - the intent is not to."}
+{"article_sent": "seven games involving nimes were investigated after conrad was arrested last november.", "correct_sent": "seven games involving nimes were investigated after conrad was arrested.", "incorrect_sent": "seven games involving nimes were investigated after conrad was investigated."}
+{"article_sent": "surrounded by her father, curt knox, mother, edda mellas and mr sutherland, knox, 27, broke down in tears as she addressed the press following italy's supreme court's ruling on friday night.", "correct_sent": "knox, 27, broke down as she addressed the press following italy's supreme court ruling.", "incorrect_sent": "curt knox, 27, broke down in tears as she addressed the press."}
+{"article_sent": "a man arrested outside a bar in austin, texas, during the sxsw (south by southwest) festival was filmed making a brazen bid for freedom -- after being handcuffed and made to lie on the floor.", "correct_sent": "a man was arrested outside a bar in austin, texas.", "incorrect_sent": "a man arrested outside a bar in austin, texas, after being arrested."}
+{"article_sent": "compound in benghazi, libya, show her aides.", "correct_sent": "attack on u.s. compound in benghazi, show her aides.", "incorrect_sent": "attack in benghazi, libya, show her aides."}
+{"article_sent": "in stuart lancaster's fourth year in charge they are yet to win this competition.", "correct_sent": "stuart lancaster's fourth year in charge they are yet to win.", "incorrect_sent": "stuart lancaster's fourth year in charge they will win this competition."}
+{"article_sent": "when the judge ordered her to stay away from her child, she seemed only happy to oblige.", "correct_sent": "judge ordered her to stay away from her child, she seemed only happy.", "incorrect_sent": "judge ordered her to stay away from her child, she said."}
+{"article_sent": "mr boon felt moved to raise the money for charity in memory of his recently deceased friends.", "correct_sent": "mr boon felt moved to charity in memory of his recently deceased friends.", "incorrect_sent": "mr boon felt the money for charity in memory of his recently deceased friends."}
+{"article_sent": "surrounded by her father, curt knox, mother, edda mellas and mr sutherland, knox, 27, broke down in tears as she addressed the press following italy's supreme court's ruling on friday night.", "correct_sent": "knox, 27, broke down in tears as she addressed the press.", "incorrect_sent": "curt knox, 27, broke down in tears as she addressed the press."}
+{"article_sent": "surrounded by her father, curt knox, mother, edda mellas and mr sutherland, knox, 27, broke down in tears as she addressed the press following italy's supreme court's ruling on friday night.", "correct_sent": "knox, 27, broke down as she addressed the press following italy's supreme court ruling.", "incorrect_sent": "knox, 27, broke down in tears after italy's supreme court's ruling."}
+{"article_sent": "she has gone on to climb africa's mount kilimanjaro, the 10 highest peaks in australia, south america's mount aconcagua in the andes, and nepal's ama dablam and manaslu.", "correct_sent": "she has gone on to climb africa's mount kilimanjaro, the 10 highest peaks in australia.", "incorrect_sent": "she has gone on to climb africa's mount kilimanjaro, the 10 highest peaks in the andes."}
+{"article_sent": "it was in march 1965 that his first hit, it's not unusual, went to no 1 in the uk and made him famous.", "correct_sent": "it was in march 1965 that his first hit, it's not unusual.", "incorrect_sent": "it was in march 1965 that his first hit, it's not famous."}
+{"article_sent": "a tweet from sportscaster fred katz challenged the today show's willie geist and uzo aduba, who plays suzanne crazy eyes warren on orange is the new black, to share a lady and the tramp licorice.", "correct_sent": "crazy eyes warren on orange is the new black", "incorrect_sent": "sportscaster fred katz challenged crazy eyes warren on orange is the new black"}
+{"article_sent": "gomis, playing his second match since collapsing at white hart lane, struck three minutes from time at villa park to prick the bubble blown by tim sherwood.", "correct_sent": "gomis struck three minutes from time at villa park.", "incorrect_sent": "gomis scored his second match since collapsing at villa park."}
+{"article_sent": "(cnn) india's daughter, the bbc four documentary about the infamous 2012 gang rape in delhi, premiered in the u.", "correct_sent": "the bbc four documentary about the infamous 2012 gang rape.", "incorrect_sent": "the bbc four documentary about the infamous 2012 gang rape in the u.s."}
+{"article_sent": "insiders say the row brought simmering tensions between the starkly contrasting pair -- both rivals for miliband's ear -- to a head.", "correct_sent": "insiders say the row brought tensions between the starkly contrasting pair.", "incorrect_sent": "insiders say row brought tensions between miliband's ear and head."}
+{"article_sent": "valentin pavlov claimed he was teaching eugene efimov how to kick the ball properly after the lad failed to show enough commitment in a tackle.", "correct_sent": "valentin pavlov claimed he was teaching eugene efimov how to kick ball properly.", "incorrect_sent": "valentin pavlov claimed he was teaching eugene efimov how to show enough commitment."}
+{"article_sent": "ex-nottingham forest boss pearce, who was in charge of england under-21s for six years and also took caretaker charge of the senior team following fabio capello's resignation, told bbc radio five live's sportsweek: 'you only really know what it's like to manage when you're in that technical area and you're the manager.", "correct_sent": "pearce has been in charge of england under-21s for six years.", "incorrect_sent": "neville has been tipped to replace hodgson's hodgson."}
+{"article_sent": "mother-of-three ayesha bhatti, 29, says she was shocked to open a bag of her favourite crisps and discovered the world's smallest wotsits inside.", "correct_sent": "ayesha bhatti, 29, says she was shocked to open bag of her favourite crisps.", "incorrect_sent": "ayesha bhatti, 29, says she was shocked to open a bag of favourite crisps."}
+{"article_sent": "mollie has already curated two edits for the high street brand and now she's back with a third.", "correct_sent": "mollie has already curated two edits for high street brand.", "incorrect_sent": "mollie has already been curated for the high street brand."}
+{"article_sent": "police have launched a probe into the 'unexplained death' of sophia walker (pictured) after she was found unresponsive in the early hours of saturday.", "correct_sent": "police have launched probe into 'unexplained death' of sophia walker.", "incorrect_sent": "police have launched a probe into the 'tragic death' of sophia walker."}
+{"article_sent": "the gunners have been on the 21-year-old's case since last season, but city are also paying close attention to his displays having impressed etihad stadium scouts in recent months.", "correct_sent": "the gunners have been on the 21-year-old's case since last season.", "incorrect_sent": "the arsenal have been on the 21-year-old's case since last season."}
+{"article_sent": "the 26-year-old began training his canine prodigy when the staffie was just 14 weeks old.", "correct_sent": "the 26-year-old began training his canine prodigy when staffie was 14 weeks old.", "incorrect_sent": "the 26-year-old began training his canine prodigy when the staffie was 14."}
+{"article_sent": "the mls outfit have released a snippet of the music video for their track 'all the way' which includes contributions by singer-songwriter sarah packiam, mc sterlin and villa.", "correct_sent": "the mls outfit have released a snippet video for their track 'all the way'", "incorrect_sent": "the mls outfit have released a music video for their track 'all the way'"}
+{"article_sent": "(cnn) the man who attacked a security area at the new orleans airport with a machete and wasp spray also had a bag of molotov cocktails and a car containing smoke bombs and gas cylinders, authorities said.", "correct_sent": "the man who attacked a security area at the new orleans airport was with a machete and a car containing smoke bombs.", "incorrect_sent": "the man who attacked a machete and wasp spray had a bag of molotov cocktails and a car containing smoke bombs."}
+{"article_sent": "kai hibbard worked on her book 'too fat, too thin, can't win' with deborah baer, according to the new york daily news.", "correct_sent": "kai hibbard worked on her book 'too fat, too thin, can't win'", "incorrect_sent": "kai hibbard worked on her book 'too fat' with deborah baer."}
+{"article_sent": "the lion died while it was being transported.", "correct_sent": "the lion lion died while it was being transported.", "incorrect_sent": "the lion died while it was being transported to hospital."}
+{"article_sent": "she has gone on to climb africa's mount kilimanjaro, the 10 highest peaks in australia, south america's mount aconcagua in the andes, and nepal's ama dablam and manaslu.", "correct_sent": "she has gone on to climb africa's mount kilimanjaro, the 10 highest peaks in australia.", "incorrect_sent": "she has gone on to climb africa's mount kilimanjaro, south america's ama dablam and manaslu."}
+{"article_sent": "manchester united have also been to watch the argentine but he is not high on their list.", "correct_sent": "manchester united have also been linked to the argentine.", "incorrect_sent": "manchester united have also been to play the argentine."}
+{"article_sent": "and a prized silver dollar each could fetch $ 10 million or more.", "correct_sent": "a silver dollar each could fetch $ 10 million or more.", "incorrect_sent": "a prized silver dollar each could fetch $ 10 or more."}
+{"article_sent": "now he has added his voice to an effort to stop the west australian government's effort to shut down as many as half of the state's 270 remote communities.", "correct_sent": "he has added his voice to an effort to shut down as many as half of the state's 270 remote communities.", "incorrect_sent": "he has added his voice to an effort to shut down as many as half of the state's 270 communities."}
+{"article_sent": "ayew has netted six times for marseille this season, as well as scoring three during the africa cup of nations.", "correct_sent": "the ayew has netted six times for marseille this season.", "incorrect_sent": "ayew has netted six during the africa cup of nations."}
+{"article_sent": "however, they were shocked when they found over 1000 pairs of runners and slippers stashed away in a 57-year-old man's melbourne unit.", "correct_sent": "they were shocked when they found over 1000 pairs of runners and slippers.", "incorrect_sent": "they were shocked when they found over 1000 pairs of runners."}
+{"article_sent": "lewis hamilton sits on the podium alongside sebastian vettel after he claimed his first victory for ferrari.", "correct_sent": "lewis hamilton claimed his first victory for ferrari.", "incorrect_sent": "lewis hamilton claimed his first win for ferrari."}
+{"article_sent": "the championship promotion hopefuls had made enquiries about signing jordan rhodes from blackburn rovers but the ewood park club's reluctance to deal has prompted fresh questions about mccormack.", "correct_sent": "the championship promotion hopefuls had made enquiries about signing jordan rhodes.", "incorrect_sent": "jordan mccormack has made enquiries about signing jordan rhodes."}
+{"article_sent": "taylor swift has been a star since the age of 16, when she released her first album and it went to no.", "correct_sent": "taylor swift has been a star since the age of 16.", "incorrect_sent": "taylor swift has been a fan since the age of 16."}
+{"article_sent": "when avery wallace was born, doctors told his parents: 'it's a girl'.", "correct_sent": "doctors told parents: 'it's a girl'", "incorrect_sent": "doctors told his parents: 'it's a."}
+{"article_sent": "mrs barnett's husband, richard, was injured in the crash and was in a stable condition in waikato hospital on thursday afternoon.", "correct_sent": "her husband, richard, was in a stable condition in waikato hospital.", "incorrect_sent": "mrs barnett was in a stable condition in waikato hospital."}
+{"article_sent": "they believe someone smashed their heads together as they stood outside the rose and crown on newgate street.", "correct_sent": "they believe someone smashed their heads together as they stood outside the rose and crown.", "incorrect_sent": "they believe someone smashed their heads together as they stood outside the crown."}
+{"article_sent": "pakistan fast bowler mohammad irfan will miss the remainder of the world cup with a pelvis injury.", "correct_sent": "mohammad irfan will miss the remainder of the world cup with a pelvis injury.", "incorrect_sent": "mohammad irfan will miss the remainder of the world cup with a pelvis."}
+{"article_sent": "the amazing collection of clips include some that have been artificially amplified to be audible to humans, and others that are simply as they originally sounded.", "correct_sent": "the collection of clips include some that have been artificially amplified to humans.", "incorrect_sent": "the amazing collection of clips include some that have been artificially audible to humans."}
+{"article_sent": "the search for noah thomas, from dublin, has widened since he was first went missing when his mother decided to get another hour of sleep on sunday morning.", "correct_sent": "the search for noah thomas has widened since he was first went missing.", "incorrect_sent": "noah thomas, from dublin, has widened since he was first went missing."}
+{"article_sent": "moores, a county championship winner with both sussex and lancashire, finds himself under increasing pressure to prove he has what it takes to succeed at international level after england's woeful showing at this world cup.", "correct_sent": "moores, a championship winner, finds himself under increasing pressure.", "incorrect_sent": "moores, a county championship, finds himself under increasing pressure."}
+{"article_sent": "ancelotti praised his side for their performance before the break in sunday's 2-0 win at home to levante but said he was concerned by the way his players relaxed and let the visitors off the hook in the second period.", "correct_sent": "ancelotti's ancelotti says he was concerned by the way his players relaxed.", "incorrect_sent": "ancelotti's side were beaten by levante in their 2-0 win over levante."}
+{"article_sent": "(cnn) india's daughter, the bbc four documentary about the infamous 2012 gang rape in delhi, premiered in the u.", "correct_sent": "the bbc four documentary about the infamous 2012 gang rape in delhi premiered.", "incorrect_sent": "the bbc four documentary about the infamous 2012 gang rape in the u.s."}
+{"article_sent": "both on screen and on the red carpet, she always radiates with a natural beauty - but while cate has avoided going under the knife or being jabbed with needles, she still relies on some beauty secrets to keep her looking her best.", "correct_sent": "cate has avoided going under the knife or being jabbed with needles.", "incorrect_sent": "she always radiates with needles, but cate has avoided going under the knife."}
+{"article_sent": "why the silence? this is not the first report to show doctors are sidestepping this tough conversation.", "correct_sent": "this is not the first report to show doctors are sidestepping this.", "incorrect_sent": "this is not the first report to show doctors this tough conversation."}
+{"article_sent": "herrera also revealed that captain wayne rooney is the joker inside united's dressing room, but admitted that he has to keep the conversation away from boxing, for fears rooney might hit him.", "correct_sent": "herrera also revealed he has to keep the conversation away from boxing.", "incorrect_sent": "herrera also revealed that rooney has to keep the conversation away from boxing."}
+{"article_sent": "it can be used with hundreds of musical apps, such as garageband because it recognises universal gestures, such as strumming, tapping, bowing and sliding, for example.", "correct_sent": "it can be used with hundreds of musical apps, such as garageband because it recognises universal gestures.", "incorrect_sent": "it can be used with hundreds of musical apps, such as garageband, tapping, bowing and sliding."}
+{"article_sent": "flare-up: election chief lucy powell, 40, left, and chief whip rosie winterton, 56, became embroiled in a dispute over ms powell's brit awards plans.", "correct_sent": "lucy powell, 40, became embroiled in a dispute over powell's brit awards plans.", "incorrect_sent": "lucy powell, 40, became embroiled in a dispute over ms powell's brit."}
+{"article_sent": "however much they deny it, men do seem to care.", "correct_sent": "however much they deny it, men do seem to care.", "incorrect_sent": "but they deny it, men do seem to care."}
+{"article_sent": "the 45-year-old replaced alex mcleish at the helm of the premier league club in june 2012 but following a poor run of results, which left villa in the bottom three, the club parted company with the scot and later appointed tim sherwood as his successor.", "correct_sent": "the 45-year-old replaced alex mcleish at the premier league in june 2012.", "incorrect_sent": "the 45-year-old replaces alex mcleish at the helm of the premier league."}
+{"article_sent": "just in june, the administration declared the success of its deal to remove al-assad's chemical weapons, which had been prompted by his august 2013 sarin attacks on syrian civilians and the threat of u.", "correct_sent": "the administration declared the success of deal to remove al-assad's chemical weapons.", "incorrect_sent": "the president declared the success of its deal to remove al-assad's chemical weapons."}
+{"article_sent": "the man, named only as saleh, said the masked militant who appears in several beheading videos was a senior figure in the extremist organisation responsible for murdering foreign captives.", "correct_sent": "the man, named only as saleh, said the masked militant who appears in beheading videos.", "incorrect_sent": "the man, named only as saleh, appears in several beheading videos."}
+{"article_sent": "atlanta falcons have been fined # 237,000 for their use of fake crowd noise at the georgia dome.", "correct_sent": "atlanta falcons fined # 237,000 for their use of fake crowd noise.", "incorrect_sent": "atlanta falcons have been fined # 237,000 for their use of fake crowd."}
+{"article_sent": "bale has received a lot of criticism, but silenced his doubters with a la liga double at the bernabeu.", "correct_sent": "bale has received a lot of criticism but silenced his doubters.", "incorrect_sent": "bale has received a lot of criticism from his la liga double."}
+{"article_sent": "the diminutive sub can dive to depths of up to 300ft (91 metres) so passengers can look at fish while staying dry and is small enough to be stowed on a yacht.", "correct_sent": "passengers can look at fish while staying dry and is small enough.", "incorrect_sent": "passengers can look at fish while staying on a yacht."}
+{"article_sent": "valentin pavlov claimed he was teaching eugene efimov how to kick the ball properly after the lad failed to show enough commitment in a tackle.", "correct_sent": "valentin pavlov claimed he was teaching eugene efimov how to kick the ball properly.", "incorrect_sent": "valentin pavlov claimed he was teaching eugene efimov how to show enough commitment."}
+{"article_sent": "but the public and commercial services union (pcs) said many are asked to go above and beyond this for no extra pay, including leading paid-for guided tours at the castle.", "correct_sent": "public and commercial services union said many are asked to go above.", "incorrect_sent": "public and commercial services union said many are asked for no extra pay."}
+{"article_sent": "the french made countless errors in dispatching italy on sunday and will have just six days to rest up for their final match, which kicks off last at 5pm and will see england knowing what is required to deliver the first notable piece of silverware of lancaster's reign.", "correct_sent": "catt bristled more at the suggestion something needs fixing.", "incorrect_sent": "scotland beat scotland in the first round of the match."}
+{"article_sent": "the 45-year-old replaced alex mcleish at the helm of the premier league club in june 2012 but following a poor run of results, which left villa in the bottom three, the club parted company with the scot and later appointed tim sherwood as his successor.", "correct_sent": "the 45-year-old replaced alex mcleish at the helm of premier league club.", "incorrect_sent": "the 45-year-old replaces alex mcleish at the helm of the premier league."}
+{"article_sent": "we reported that, just before we battened down the hatches, all the three and four-year-old boys had been snapping at their pyjama elastic and comparing the size of their willies.", "correct_sent": "the three and four-year-old boys had been snapping at their pyjama elastic.", "incorrect_sent": "the three and boys had been snapping at their pyjama elastic."}
+{"article_sent": "a man arrested outside a bar in austin, texas, during the sxsw (south by southwest) festival was filmed making a brazen bid for freedom -- after being handcuffed and made to lie on the floor.", "correct_sent": "a man arrested outside a bar in austin, texas, on the floor.", "incorrect_sent": "a man arrested outside a bar in austin, texas, after being arrested."}
+{"article_sent": "while it doesn't exactly fly the flag for sexual equality, the annual wife carrying world championships certainly appears to be a fun couple of days in sonkaj\u00e4rvi, finland.", "correct_sent": "the annual wife carrying world championships appears to be a fun couple of days.", "incorrect_sent": "the annual wife carrying world championships appears to be a fun couple."}
+{"article_sent": "the driver's side of the windscreen immediately shatters and falls through.", "correct_sent": "driver's side of windscreen immediately shatters and falls through.", "incorrect_sent": "driver's side were immediately shatters and falls through."}
+{"article_sent": "mersinger says dube covered up nichole's body with leaves and sticks.", "correct_sent": "mersinger says dube covered up nichole's body with leaves and sticks.", "incorrect_sent": "dube says dube covered up nichole's body with leaves and sticks."}
+{"article_sent": "aubrey said she was also told her son would be eligible for a wheelchair when he is five years old.", "correct_sent": "aubrey said her son would be eligible for a wheelchair when he is five years old.", "incorrect_sent": "aubrey said she was eligible for a wheelchair when he is five years old."}
+{"article_sent": "we were left with a cryptic reference to the news that jean had died in asuncion, paraguay.", "correct_sent": "jean had died in asuncion, paraguay.", "incorrect_sent": "the news that jean had died in asuncion, paraguay."}
+{"article_sent": "the thousands of chickens live together in huge sheds for just 35 days before they are gassed.", "correct_sent": "the chickens live together in huge sheds for just 35 days before they are gassed.", "incorrect_sent": "the thousands live together in huge sheds for just 35 days before they are gassed."}
+{"article_sent": "passengers face travel chaos this easter as britain's busiest rail routes are expected to launch major engineering projects (file picture).", "correct_sent": "britain's busiest rail routes are expected to launch major engineering projects.", "incorrect_sent": "passengers face travel chaos this easter as britain's busiest rail routes."}
+{"article_sent": "if the falcons have multiple picks in that round, the highest selection will be forfeited.", "correct_sent": "if the game, the highest selection will be forfeited.", "incorrect_sent": "if the game, the highest selection will be forfeited."}
+{"article_sent": "russell brand (left) has admitted to getting treatment for sex addiction and ulrika jonsson (right) presented a 2007 channel 4 show called am i a sex addict? in which she questioned her own romantic behaviour.", "correct_sent": "admitted to having treatment for sex addiction and ulrika jonsson.", "incorrect_sent": "admitted to getting treatment for sex addiction and jonsson jonsson."}
+{"article_sent": "outside, it's a typical day -- cold and damp, with a sky as grey and heavy as old army boots.", "correct_sent": "it's a typical day -- cold and damp, with a sky as old army boots.", "incorrect_sent": "it's a typical day -- cold and damp, with sky as old army boots."}
+{"article_sent": "seven games involving nimes were investigated after conrad was arrested last november.", "correct_sent": "seven games were investigated after conrad was arrested last november.", "incorrect_sent": "seven games involving nimes were arrested last november."}
+{"article_sent": "burnley crown court was told the couple never lived together but would see each other several times a week and she had keys to his property.", "correct_sent": "couple never lived together but would see each other times a week.", "incorrect_sent": "burnley crown court told the couple never lived together."}
+{"article_sent": "the current generation of young people is the most confused about sex ever because of pornography, according to a psychotherapist.", "correct_sent": "the current generation of young people is the most confused about sex.", "incorrect_sent": "the current generation of young people is the most confused about pornography."}
+{"article_sent": "migrants who come to britain to live or work must prove they have not committed a crime in the previous ten years under new measures to tighten up the nation's borders.", "correct_sent": "migrants who come to britain to live or work must prove they have not committed crime.", "incorrect_sent": "migrants who come to britain to live or work must prove they have committed a crime."}
+{"article_sent": "mollie has already curated two edits for the high street brand and now she's back with a third.", "correct_sent": "mollie has curated two edits for the high street brand.", "incorrect_sent": "mollie has already been curated for the high street brand."}
+{"article_sent": "(cnn) spirit airlines has a new fare deal and is announcing it with sex jokes.", "correct_sent": "spirit airlines has new fare deal with sex jokes.", "incorrect_sent": "spirit airlines has a new fare deal with sex."}
+{"article_sent": "the home which was built for former australian prime minister malcolm fraser and his wife tamie has been opened for inspection just a day after his sudden passing.", "correct_sent": "the home was built for former australian pm malcolm fraser and his wife tamie.", "incorrect_sent": "the home was built for inspection, just a day after his sudden passing."}
+{"article_sent": "my great-great-grandfather, jean bourdain, was one of them.", "correct_sent": "my great-great-grandfather, jean bourdain was one of them.", "incorrect_sent": "my great-great-grandfather, jean bourdain, is one of them."}
+{"article_sent": "lewis hamilton sits on the podium alongside sebastian vettel after he claimed his first victory for ferrari.", "correct_sent": "lewis hamilton claimed his first victory for ferrari.", "incorrect_sent": "lewis hamilton claimed his first victory for his ferrari."}
+{"article_sent": "however, there is also interest from roma while tottenham and everton have asked to be kept informed.", "correct_sent": "tottenham and everton have asked to be kept informed.", "incorrect_sent": "roma and tottenham have asked to be kept informed."}
+{"article_sent": "the current generation of young people is the most confused about sex ever because of pornography, according to a psychotherapist.", "correct_sent": "the current generation is the most confused about sex ever because of pornography.", "incorrect_sent": "the current generation of young people is the most confused about pornography."}
+{"article_sent": "rima karaki was conducting an interview with london-based islamist hani al-seba' i for lebanon's al jadeed on the weekend of international women's day, discussing reports of christians joining islamic groups such as isis in the middle east when the scholar went off on a historical tangent.", "correct_sent": "rima karaki was conducting an interview with islamist hani al-seba.", "incorrect_sent": "rima karaki was conducting an interview with london-based islamist al-seba al-seba."}
+{"article_sent": "england coach peter moores talks to the media during a press conference at the adelaide oval on sunday.", "correct_sent": "coach peter moores talks to the media during a press conference.", "incorrect_sent": "coach peter moores talks to the adelaide oval on sunday."}
+{"article_sent": "the home which was built for former australian prime minister malcolm fraser and his wife tamie has been opened for inspection just a day after his sudden passing.", "correct_sent": "the home was built for former prime minister malcolm fraser and his wife tamie.", "incorrect_sent": "the home was built for inspection just a day after his sudden passing."}
+{"article_sent": "the 21-year-old is valued at # 30million by palermo president mauro zamparini and juventus are considered favourites even though they have had an opening offer rejected.", "correct_sent": "the 21-year-old is valued at # 30million by palermo president mauro zamparini.", "incorrect_sent": "the 21-year-old is valued at # 30million for palermo president mauro zamparini."}
+{"article_sent": "families reacted with fury yesterday at learning that tony blair will attend a service honouring soldiers killed in afghanistan.", "correct_sent": "tony blair will attend a service honouring soldiers killed in afghanistan.", "incorrect_sent": "families reacted with fury yesterday at learning that tony blair killed in afghanistan."}
+{"article_sent": "however, they were shocked when they found over 1000 pairs of runners and slippers stashed away in a 57-year-old man's melbourne unit.", "correct_sent": "they were shocked when they found over 1000 pairs of runners and slippers stashed away.", "incorrect_sent": "they were shocked when they found over 1000 pairs of runners."}
+{"article_sent": "louis van gaal said that he believed wenger's estimation of the necessary points total.", "correct_sent": "louis van gaal says he believed wenger's estimation of the points total.", "incorrect_sent": "louis van gaal says wenger's estimation of the necessary points total."}
+{"article_sent": "the british tennis player left red-faced when andy murray claimed he had a 'little girlfriend in glasgow' - as well as his long-term partner in london - has insisted it was all a joke.", "correct_sent": "andy murray has insisted it was all a joke.", "incorrect_sent": "andy murray has insisted it was all a joke in glasgow."}
+{"article_sent": "the british no 1 went down 6-2, 6-3 in 88 minutes to his direct contemporary on saturday in the latest match of what has begun to look like a one-sided rivalry.", "correct_sent": "british no 1 went down 6-2, 6-3 in 88 minutes.", "incorrect_sent": "british no 1 went down 6-2 in 88 minutes to his direct contemporary."}
+{"article_sent": "kai hibbard worked on her book 'too fat, too thin, can't win' with deborah baer, according to the new york daily news.", "correct_sent": "kai hibbard worked on her book 'too fat, can't win' with deborah baer.", "incorrect_sent": "kai hibbard worked on her book 'too fat' with deborah baer."}
+{"article_sent": "now, though, kane is one of the most talked about players in the country as his roy of the rovers rise continues apace.", "correct_sent": "kane is one of the most talked about players in the country.", "incorrect_sent": "kane is one of the most talked about players in the rovers."}
+{"article_sent": "an unconscious patient secretly raped in a&e by a male nurse who filmed the attack admitted today she tried to kill herself after police revealed her ordeal.", "correct_sent": "she was secretly raped by a male nurse who filmed the attack.", "incorrect_sent": "police say she tried to kill herself after police revealed her ordeal."}
+{"article_sent": "the home which was built for former australian prime minister malcolm fraser and his wife tamie has been opened for inspection just a day after his sudden passing.", "correct_sent": "the home was built for former prime minister malcolm fraser and his wife tamie.", "incorrect_sent": "the home was built for inspection, just a day after his sudden passing."}
+{"article_sent": "katie launched an attack on danny's partner joanne mas, saying: 'no woman should ever propose to a man,' to which the eastenders star responded:' i agree.", "correct_sent": "katie launched an attack on danny's partner joanne mas.", "incorrect_sent": "katie launched an attack on danny mas, saying: 'no woman should ever propose to a man'"}
+{"article_sent": "jacqueline crane, 52, of of st austell, cornwall, pretended to be a professor of psychology and claimed she was qualified to train people to become counsellors.", "correct_sent": "jacqueline crane, 52, pretended she was qualified to train people to become counsellors.", "incorrect_sent": "jacqueline crane, 52, pretended to be a professor of st austell, cornwall."}
+{"article_sent": "the amazing collection of clips include some that have been artificially amplified to be audible to humans, and others that are simply as they originally sounded.", "correct_sent": "the amazing collection of clips include some that have been artificially amplified to humans.", "incorrect_sent": "amazing collection of clips include some that have been artificially amplified to humans."}
+{"article_sent": "a jury at wolverhampton crown court heard how the duo had planned to carry out a robbery via text messages a fortnight before the murder.", "correct_sent": "jury heard how the pair had planned to carry out a robbery via text messages.", "incorrect_sent": "jury told the duo had planned to carry out a robbery via text messages."}
+{"article_sent": "some surgeries in cumbria are so desperate for staff they are paying locum gps # 850 a day to stand in, a new bbc inside out documentary has discovered.", "correct_sent": "some surgeries in cumbria are so desperate for staff they are paying locum gps # 850 a day to stand.", "incorrect_sent": "some surgeries in cumbria are so desperate to stand in, a new bbc inside out documentary has discovered."}
+{"article_sent": "' sex addiction is described by the relationship counselling service relate as any sexual activity that feels out of control.", "correct_sent": "sex addiction is also described by the relationship counselling service relate.", "incorrect_sent": "sex service relate to sexual activity that feels out of control."}
+{"article_sent": "the scotland international made an # 11million move from leeds in the summer but the move has backfired with fulham falling perilously close to the relegation places in recent weeks.", "correct_sent": "scotland international made an # 11million move from leeds in the summer.", "incorrect_sent": "scotland international made an # 11million move from leeds in recent weeks."}
+{"article_sent": "seven games involving nimes were investigated after conrad was arrested last november.", "correct_sent": "seven games involving nimes were investigated after conrad's arrest.", "incorrect_sent": "seven games involving nimes were arrested last november."}
+{"article_sent": "compound in benghazi, libya, show her aides.", "correct_sent": "attack on u.s. compound in benghazi, libya.", "incorrect_sent": "attack in benghazi, libya, show her aides."}
+{"article_sent": "paulo dybala of palermo is a transfer target for three british clubs in the summer, but may cost # 30million.", "correct_sent": "paulo dybala is a transfer target for three british clubs in the summer.", "incorrect_sent": "palermo is a transfer target for three british clubs in the summer."}
+{"article_sent": "fulham striker mccormack gets his shot away as fulham take on watford in the championship.", "correct_sent": "fulham striker mccormack gets shot away as fulham take on watford.", "incorrect_sent": "fulham striker mccormack gets his shot away as fulham win on watford."}
+{"article_sent": "police have been forced to play prince charming after numerous residents complained that their shoes had gone missing from their front gardens.", "correct_sent": "police have been forced to play prince charming after numerous complaints.", "incorrect_sent": "police have complained that their shoes had gone missing from their front gardens."}
+{"article_sent": "a jury at wolverhampton crown court heard how the duo had planned to carry out a robbery via text messages a fortnight before the murder.", "correct_sent": "jury heard how the duo had planned to carry out a robbery via text messages.", "incorrect_sent": "jury told the duo had planned to carry out a robbery via text messages."}
+{"article_sent": "former manchester city boss roberto mancini has turned up the heat on successor manuel pellegrini by claiming the chilean should be winning a trophy every year.", "correct_sent": "roberto mancini has turned up the heat on successor manuel pellegrini.", "incorrect_sent": "roberto mancini has turned up the heat for manuel pellegrini's successor."}
+{"article_sent": "louis van gaal said that he believed wenger's estimation of the necessary points total.", "correct_sent": "louis van gaal says he believed wenger is estimation of the necessary points.", "incorrect_sent": "louis van gaal says wenger's estimation of the necessary points total."}
+{"article_sent": "lloris wearing the scarf after defeat by chelsea in the capital one cup final at wembley.", "correct_sent": "lloris wearing the scarf after defeat by chelsea in the capital.", "incorrect_sent": "lloris wearing the scarf after defeat by chelsea in the fa cup final."}
+{"article_sent": "in an anonymous confession to australia's gold coast bulletin the woman reveals:' i once had sex with two guys in one night before going to meet up with the guy i was supposed to be seeing at the time.", "correct_sent": "the woman reveals she once had sex with two guys in one night.", "incorrect_sent": "the woman reveals she was supposed to be seeing the woman at the time."}
+{"article_sent": "he was also ordered to pay $ 167 million in restitution and fines.", "correct_sent": "he also ordered to pay $ 167 million in restitution and fines.", "incorrect_sent": "he was also ordered to pay $ 167 million in damages and fines."}
diff --git a/langtest/data/config/factuality_huggingface_config.yml b/langtest/data/config/factuality_huggingface_config.yml
new file mode 100644
index 000000000..f851aff76
--- /dev/null
+++ b/langtest/data/config/factuality_huggingface_config.yml
@@ -0,0 +1,11 @@
+model_parameters:
+  temperature: 0.2
+  max_new_tokens: 62
+
+tests:
+  defaults:
+    min_pass_rate: 0.80
+
+  factuality:
+    order_bias:
+      min_pass_rate: 0.70
diff --git a/langtest/data/config/factuality_openai_config.yml b/langtest/data/config/factuality_openai_config.yml
new file mode 100644
index 000000000..4ede248f4
--- /dev/null
+++ b/langtest/data/config/factuality_openai_config.yml
@@ -0,0 +1,7 @@
+tests:
+  defaults:
+    min_pass_rate: 0.80
+
+  factuality:
+    order_bias:
+      min_pass_rate: 0.70
\ No newline at end of file
diff --git a/langtest/datahandler/datasource.py b/langtest/datahandler/datasource.py
index 516c58c06..5d0ef1d57 100644
--- a/langtest/datahandler/datasource.py
+++ b/langtest/datahandler/datasource.py
@@ -30,6 +30,7 @@
     DisinformationSample,
     WinoBiasSample,
     LegalSample,
+    FactualitySample,
 )
 from ..utils.lib_manager import try_import_lib
 
@@ -81,6 +82,11 @@
         "legal_conclusion_b": ["legal_conclusion_b"],
         "correct_choice": ["correct_choice"],
     },
+    "factuality-test": {
+        "article_sent": ["article_sent"],
+        "correct_sent": ["correct_sent"],
+        "incorrect_sent": ["incorrect_sent"],
+    },
 }
 
 
@@ -312,6 +318,8 @@ def _load_dataset(cls, dataset_name: str) -> str:
             + "/NarrativeWedging/Narrative_Wedging.jsonl",
             "Wino-test": script_dir[:-7] + "/Wino-Bias/wino-bias-test.jsonl",
             "Legal-Support-test": script_dir[:-7] + "/Legal-Support/legal-test.jsonl",
+            "Factual-Summary-Pairs": script_dir[:-7]
+            + "/Factuality/Factual-Summary-Pairs.jsonl",
         }
 
         return datasets_info[dataset_name]
@@ -1129,6 +1137,7 @@ class JSONLDataset(_IDataset):
         "disinformation-test",
         "wino-bias",
         "legal-tests",
+        "factuality-test",
     ]
     COLUMN_NAMES = {task: COLUMN_MAPPER[task] for task in supported_tasks}
 
@@ -1290,6 +1299,15 @@ def load_data(self) -> List[Sample]:
                         )
                     )
 
+                elif self.task == "factuality-test":
+                    data.append(
+                        FactualitySample(
+                            article_sent=item["article_sent"],
+                            incorrect_sent=item["incorrect_sent"],
+                            correct_sent=item["correct_sent"],
+                            dataset_name=self._file_path.split("/")[-2],
+                        )
+                    )
         return data
 
     def export_data(self, data: List[Sample], output_path: str):
diff --git a/langtest/langtest.py b/langtest/langtest.py
index ce96325dc..9c2675be1 100644
--- a/langtest/langtest.py
+++ b/langtest/langtest.py
@@ -43,6 +43,7 @@ class Harness:
         "political",
         "wino-bias",
         "legal-tests",
+        "factuality-test",
     ]
     SUPPORTED_HUBS = [
         "spacy",
@@ -106,6 +107,15 @@ class Harness:
             "disinformation-test-ai21": resource_filename(
                 "langtest", "data/config/disinformation_openai_config.yml"
             ),
+            "factuality-test-huggingface-inference-api": resource_filename(
+                "langtest", "data/config/factuality_huggingface_config.yml"
+            ),
+            "factuality-test-openai": resource_filename(
+                "langtest", "data/config/factuality_openai_config.yml"
+            ),
+            "factuality-test-ai21": resource_filename(
+                "langtest", "data/config/factuality_openai_config.yml"
+            ),
             "translation-huggingface": resource_filename(
                 "langtest", "data/config/translation_transformers_config.yml"
             ),
@@ -253,7 +263,7 @@ def __init__(
         elif hub in self.DEFAULTS_CONFIG["hubs"]:
             if task in self.DEFAULTS_CONFIG["task"]:
                 self._config = self.configure(self.DEFAULTS_CONFIG["task"][task])
-            elif task == "disinformation-test":
+            elif task in ["disinformation-test", "factuality-test"]:
                 self._config = self.configure(
                     self.DEFAULTS_CONFIG["task"][task + "-" + hub]
                 )
@@ -933,6 +943,11 @@ def generated_results(self) -> Optional[pd.DataFrame]:
             "completion_toxicity",
             "hypothesis",
             "statements",
+            "article_sentence",
+            "correct_sentence",
+            "incorrect_sentence",
+            "result",
+            "swapped_result",
             "model_response",
             "eval_score",
             "similarity_score",
@@ -1086,6 +1101,9 @@ def testcases(self) -> pd.DataFrame:
             "diagnosis",
             "hypothesis",
             "statements",
+            "article_sentence",
+            "correct_sentence",
+            "incorrect_sentence",
             "perturbed_context",
             "perturbed_question",
             "expected_result",
diff --git a/langtest/modelhandler/llm_modelhandler.py b/langtest/modelhandler/llm_modelhandler.py
index d4edf2a50..214900e95 100644
--- a/langtest/modelhandler/llm_modelhandler.py
+++ b/langtest/modelhandler/llm_modelhandler.py
@@ -220,3 +220,13 @@ class PretrainedModelForLegal(PretrainedModelForQA, _ModelHandler):
     """
 
     pass
+
+
+class PretrainedModelForFactualityTest(PretrainedModelForQA, _ModelHandler):
+    """A class representing a pretrained model for factuality detection.
+
+    Inherits:
+        PretrainedModelForQA: The base class for pretrained models.
+    """
+
+    pass
diff --git a/langtest/modelhandler/modelhandler.py b/langtest/modelhandler/modelhandler.py
index 194ae5565..9f8887c8b 100644
--- a/langtest/modelhandler/modelhandler.py
+++ b/langtest/modelhandler/modelhandler.py
@@ -56,6 +56,7 @@ class ModelFactory:
         "political",
         "wino-bias",
         "legal-tests",
+        "factuality-test",
     ]
     SUPPORTED_MODULES = [
         "pyspark",
@@ -165,6 +166,11 @@ def __init__(self, model: str, task: str, hub: str, *args, **kwargs):
         elif task == "wino-bias":
             self.model_class = model_handler.PretrainedModelForWinoBias(model)
 
+        elif task == "factuality-test":
+            self.model_class = model_handler.PretrainedModelForFactualityTest(
+                hub, model, *args, **kwargs
+            )
+
         else:
             self.model_class = model_handler.PretrainedModelForTextClassification(model)
 
@@ -278,9 +284,15 @@ def load_model(
                     hub, path, *args, **kwargs
                 )
             )
+
         elif task in ("wino-bias"):
             model_class = modelhandler_module.PretrainedModelForWinoBias.load_model(path)
 
+        elif task in ("factuality-test"):
+            model_class = modelhandler_module.PretrainedModelForFactualityTest.load_model(
+                hub, path, *args, **kwargs
+            )
+
         else:
             model_class = (
                 modelhandler_module.PretrainedModelForTextClassification.load_model(path)
diff --git a/langtest/modelhandler/transformers_modelhandler.py b/langtest/modelhandler/transformers_modelhandler.py
index f8c22378b..c43c5023f 100644
--- a/langtest/modelhandler/transformers_modelhandler.py
+++ b/langtest/modelhandler/transformers_modelhandler.py
@@ -507,3 +507,22 @@ class PretrainedModelForPolitical(PretrainedModelForQA, _ModelHandler):
     """
 
     pass
+
+
+class PretrainedModelForDisinformationTest(PretrainedModelForQA, _ModelHandler):
+    """A class representing a pretrained model for disinformation test.
+    Inherits:
+        PretrainedModelForQA: The base class for pretrained models.
+    """
+
+    pass
+
+
+class PretrainedModelForFactualityTest(PretrainedModelForQA, _ModelHandler):
+    """A class representing a pretrained model for factuality detection.
+
+    Inherits:
+        PretrainedModelForQA: The base class for pretrained models.
+    """
+
+    pass
diff --git a/langtest/transform/__init__.py b/langtest/transform/__init__.py
index b76d33b42..988d8bed9 100644
--- a/langtest/transform/__init__.py
+++ b/langtest/transform/__init__.py
@@ -1762,3 +1762,83 @@ async def async_run(sample_list: List[Sample], model: ModelFactory, *args, **kwa
             if progress:
                 progress.update(1)
         return sample_list["legal-support"]
+
+
+class FactualityTestFactory(ITests):
+    """Factory class for factuality test"""
+
+    alias_name = "factuality"
+    supported_tasks = [
+        "factuality-test",
+    ]
+
+    def __init__(self, data_handler: List[Sample], tests: Dict = None, **kwargs) -> None:
+        """Initializes the factuality-test"""
+        self.data_handler = data_handler
+        self.tests = tests
+        self.kwargs = kwargs
+
+    def transform(self) -> List[Sample]:
+        """Nothing to use transform for no longer to generating testcases.
+
+        Returns:
+            Empty list
+
+        """
+        for sample in self.data_handler:
+            sample.test_type = "order_bias"
+            sample.category = "factuality"
+
+        return self.data_handler
+
+    @classmethod
+    async def run(
+        cls, sample_list: List[Sample], model: ModelFactory, **kwargs
+    ) -> List[Sample]:
+        """Runs factuality tests
+
+        Args:
+            sample_list (list[Sample]): A list of Sample objects to be tested.
+            model (ModelFactory): The model to be used for evaluation.
+            **kwargs: Additional keyword arguments.
+
+        Returns:
+            list[Sample]: A list of Sample objects with test results.
+
+        """
+        task = asyncio.create_task(cls.async_run(sample_list, model, **kwargs))
+        return task
+
+    @classmethod
+    def available_tests(cls) -> Dict[str, str]:
+        """Retrieves available factuality test types.
+
+        Returns:
+            dict: A dictionary mapping test names to their corresponding classes.
+
+        """
+        return {"order_bias": cls}
+
+    async def async_run(sample_list: List[Sample], model: ModelFactory, *args, **kwargs):
+        """Runs factuality tests
+
+        Args:
+            sample_list (list[Sample]): A list of Sample objects to be tested.
+            model (ModelFactory): The model to be used for testing.
+            *args: Additional positional arguments.
+            **kwargs: Additional keyword arguments.
+
+        Returns:
+            list[Sample]: A list of Sample objects with test results.
+
+        """
+        progress = kwargs.get("progress_bar", False)
+        for sample in sample_list["order_bias"]:
+            if sample.state != "done":
+                if hasattr(sample, "run"):
+                    sample_status = sample.run(model, **kwargs)
+                    if sample_status:
+                        sample.state = "done"
+            if progress:
+                progress.update(1)
+        return sample_list["order_bias"]
diff --git a/langtest/utils/custom_types/__init__.py b/langtest/utils/custom_types/__init__.py
index 9725f99bf..53345af29 100644
--- a/langtest/utils/custom_types/__init__.py
+++ b/langtest/utils/custom_types/__init__.py
@@ -16,6 +16,7 @@
     DisinformationSample,
     WinoBiasSample,
     LegalSample,
+    FactualitySample,
 )
 from .helpers import Span, Transformation
 from .output import (
diff --git a/langtest/utils/custom_types/helpers.py b/langtest/utils/custom_types/helpers.py
index 5fd44fad9..6728d63b6 100644
--- a/langtest/utils/custom_types/helpers.py
+++ b/langtest/utils/custom_types/helpers.py
@@ -54,6 +54,21 @@
 
     Choose the legal conclusion that most forcefully and directly supports the legal claim in the passage. If it is conclusion A, response must be just "a", else "b".
     """,
+    "factuality": """Decide which of the following summary is more consistent with the article sentence.
+
+     Note that consistency means
+     all information in the summary is supported by the article.
+
+     Do not bias towards the first summary.
+
+     Do not bias towards the second summary.
+
+     Both answers are equally probable.
+
+     Article Sentence: {article_sentence}
+     Summary A: {option_a}
+     Summary B: {option_b}
+     Answer (A or B):""",
 }
 
 
diff --git a/langtest/utils/custom_types/sample.py b/langtest/utils/custom_types/sample.py
index 155b874cb..9c62167e0 100644
--- a/langtest/utils/custom_types/sample.py
+++ b/langtest/utils/custom_types/sample.py
@@ -1,3 +1,5 @@
+import re
+import string
 from typing import Any, Dict, List, Optional, Tuple, TypeVar, Union, Callable
 from copy import deepcopy
 from pydantic import BaseModel, PrivateAttr, validator, Field
@@ -1447,6 +1449,254 @@ def run(self, model, **kwargs):
         return True
 
 
+class FactualitySample(BaseModel):
+    """
+    A class representing a sample for the Factuality task.
+
+    Attributes:
+        article_sent (str): The original article sentence.
+        incorrect_sent (str): The incorrect version of the sentence.
+        correct_sent (str): The correct version of the sentence.
+        state (str, optional): The state of the sample (e.g., 'draft', 'final').
+        dataset_name (str, optional): The name of the dataset.
+        task (str, optional): The task related to the sample.
+        category (str, optional): The category of the sample.
+        test_type (str, optional): The type of test conducted on the sample.
+        result (str, optional): Stores the output when the correct summary is presented first.
+        swapped_result (str, optional): Stores the output when the incorrect summary is presented first.
+
+    Methods:
+        to_dict(): Convert the sample to a dictionary.
+        is_pass(): Check if the sample passes the evaluation.
+        remove_punctuation(input_string): Remove punctuation from the input string.
+        _is_eval(): Internal method to evaluate the sample.
+        run(model, **kwargs): Run the sample through a specified model.
+    """
+
+    article_sent: str
+    incorrect_sent: str
+    correct_sent: str
+    state: str = None
+    dataset_name: str = None
+    task: str = None
+    category: str = None
+    test_type: str = None
+    result: str = None
+    swapped_result: str = None
+
+    def __init__(self, **data):
+        super().__init__(**data)
+
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Convert the sample to a dictionary.
+
+        Returns:
+            dict: A dictionary representation of the sample.
+        """
+        result = {
+            "article_sentence": self.article_sent,
+            "correct_sentence": self.correct_sent,
+            "incorrect_sentence": self.incorrect_sent,
+            "category": self.category,
+            "test_type": self.test_type,
+        }
+
+        if self.result is not None and self.swapped_result is not None:
+            bool_pass = self._is_eval()
+            result.update(
+                {
+                    "result": self.result,
+                    "swapped_result": self.swapped_result,
+                    "pass": bool_pass,
+                }
+            )
+
+        return result
+
+    def is_pass(self):
+        """
+        Check if the sample passes the evaluation.
+
+        Returns:
+            bool: True if the sample passes, False otherwise.
+        """
+        return self._is_eval()
+
+    def remove_punctuation(self, input_string):
+        """
+        Remove punctuation from the input string.
+
+        Args:
+            input_string (str): The input string with punctuation.
+
+        Returns:
+            str: The input string with punctuation removed.
+        """
+        translator = str.maketrans("", "", string.punctuation)
+
+        cleaned_string = input_string.translate(translator)
+
+        return cleaned_string
+
+    def _is_eval(self) -> bool:
+        """
+        Internal method to evaluate the sample.
+
+        Returns:
+            bool: True if the sample passes, False otherwise.
+        """
+        R1 = False
+        R2 = False
+        valid_results = ("A", "B", "a", "b", "ab", "ba")
+        self.result = self.result.strip()
+        self.swapped_result = self.swapped_result.strip()
+
+        pattern_a = re.compile(r"(Answer A|Summary A)", re.IGNORECASE)
+        pattern_b = re.compile(r"(Answer B|Summary B)", re.IGNORECASE)
+        pattern_ab = re.compile(
+            r"(Answer (A or B|B or A|A and B|B and A)|"
+            r"Summary (A or B|B or A|A and B|B and A)|"
+            r"Both (A and B|B and A|A or B|B or A|A B|B A)|Both)",
+            re.IGNORECASE,
+        )
+        extra_check_a = re.compile(r"^(A[.?!,:]|A\n)", re.IGNORECASE)
+        extra_check_b = re.compile(r"^(B[.?!,:]|B\n)", re.IGNORECASE)
+
+        if (
+            "".join(filter(str.isalnum, self.result)) in valid_results
+            and "".join(filter(str.isalnum, self.swapped_result)) in valid_results
+        ):
+            if (
+                "".join(filter(str.isalnum, self.result)).lower() == "a"
+                and "".join(filter(str.isalnum, self.swapped_result)).lower() == "b"
+            ):
+                return True
+            elif "".join(filter(str.isalnum, self.result)) in ["ab", "ba"] or "".join(
+                filter(str.isalnum, self.swapped_result)
+            ) in ["ab", "ba"]:
+                return False
+            else:
+                return False
+        else:
+            if (
+                (
+                    pattern_ab.search(self.remove_punctuation(self.result))
+                    or pattern_ab.search(self.remove_punctuation(self.swapped_result))
+                )
+                or (
+                    pattern_a.search(self.remove_punctuation(self.result))
+                    and pattern_b.search(self.remove_punctuation(self.result))
+                )
+                or (
+                    pattern_a.search(self.remove_punctuation(self.swapped_result))
+                    and pattern_b.search(self.remove_punctuation(self.swapped_result))
+                )
+            ):
+                return False
+            if (
+                "".join(filter(str.isalnum, self.result)).lower() == "a"
+                or pattern_a.search(self.remove_punctuation(self.result))
+                or extra_check_a.search(self.result)
+            ):
+                R1 = True
+            if (
+                "".join(filter(str.isalnum, self.swapped_result)).lower() == "b"
+                or pattern_b.search(self.remove_punctuation(self.swapped_result))
+                or extra_check_b.search(self.swapped_result)
+            ):
+                R2 = True
+            if (
+                "".join(filter(str.isalnum, self.result)).lower() == "b"
+                or pattern_b.search(self.remove_punctuation(self.result))
+                or extra_check_b.search(self.result)
+            ):
+                return False
+            if (
+                "".join(filter(str.isalnum, self.swapped_result)).lower() == "a"
+                or pattern_a.search(self.remove_punctuation(self.swapped_result))
+                or extra_check_a.search(self.swapped_result)
+            ):
+                return False
+
+            if R1 and R2:
+                return True
+
+            else:
+                from ...langtest import HARNESS_CONFIG as harness_config
+
+                config = harness_config["tests"]["defaults"]
+
+                from ..SentenceTransformer import SimpleSentenceTransformer
+
+                model = SimpleSentenceTransformer(
+                    model_name="sentence-transformers/distiluse-base-multilingual-cased-v2"
+                )
+
+                threshold = config.get("threshold", 0.85)
+
+                if R1:
+                    embeddings2 = model.encode([self.swapped_result, self.correct_sent])
+                    similarity2 = cosine_similarity([embeddings2[0]], [embeddings2[1]])[0]
+                    return similarity2 > threshold
+
+                elif R2:
+                    embeddings1 = model.encode([self.result, self.correct_sent])
+                    similarity1 = cosine_similarity([embeddings1[0]], [embeddings1[1]])[0]
+                    return similarity1 > threshold
+
+                else:
+                    embeddings1 = model.encode([self.result, self.correct_sent])
+                    similarity1 = cosine_similarity([embeddings1[0]], [embeddings1[1]])[0]
+                    embeddings2 = model.encode([self.swapped_result, self.correct_sent])
+                    similarity2 = cosine_similarity([embeddings2[0]], [embeddings2[1]])[0]
+
+                    return all(
+                        similarity > threshold
+                        for similarity in [similarity1, similarity2]
+                    )
+
+    def run(self, model, **kwargs):
+        """
+        Run the sample through a specified model.
+
+        Args:
+            model: The machine learning model to run the sample through.
+            **kwargs: Additional keyword arguments.
+
+        Returns:
+            bool: True if the operation was successful.
+        """
+        dataset_name = self.dataset_name.split("-")[0].lower()
+        prompt_template = kwargs.get(
+            "user_prompt",
+            default_user_prompt.get(dataset_name, ""),
+        )
+        self.result = model(
+            text={
+                "article_sentence": self.article_sent,
+                "option_a": self.correct_sent,
+                "option_b": self.incorrect_sent,
+            },
+            prompt={
+                "template": prompt_template,
+                "input_variables": ["article_sentence", "option_a", "option_b"],
+            },
+        )
+        self.swapped_result = model(
+            text={
+                "article_sentence": self.article_sent,
+                "option_a": self.incorrect_sent,
+                "option_b": self.correct_sent,
+            },
+            prompt={
+                "template": prompt_template,
+                "input_variables": ["article_sentence", "option_a", "option_b"],
+            },
+        )
+        return True
+
+
 Sample = TypeVar(
     "Sample",
     MaxScoreSample,
@@ -1455,4 +1705,5 @@ def run(self, model, **kwargs):
     NERSample,
     SummarizationSample,
     LLMAnswerSample,
+    FactualitySample,
 )