## USER
You are a university professor and eloquent creative writing coauthor who has many books on the philosophy of human thought, creativity and technology. We are going to write an article focused on an audience of academics working in digital humanities, in response to a number of concerns they have about LLMs and ChatGPT. The essay is a response to the speech of Bernardo Bueno, who has brought up some points about ChatGPT and its (mis)use in creative writing. I wish to cover the following items succinctly and in detail, leading up to a guide to creative prompt engineering:

Electronic - Generative - Interactive writing → ChatGPT

Issues:

- Lack of knowledge on how it works
- Size vs Purpose
- Too much knowledge and power in the hands of few (companies) - once it’s out there, it’s out there = consent?
- from Social Media: Fun, realization of harm to mental health and self, lack of regulation, use by children
- Plagiarism and Fraud - ethics
- Are humans obsolete?

Human/Machine/Hybrid creation

The metal and wooden Theseus’ ship - how different is human and AI creativity? Where does one become the other

Questions

- Virtual IS real
- Technology IS Human
- Tools and Ethics Training
- Simulation and Creation
- The role of Digital Humanities

I wish to add these too:

- any metaphors that could help in understanding or visualizing the ideas.
- imagined fears, why they happen, and some examples of them when it comes to ChatGPT (e.g. AGI, the uncanny valley, human and job obsolescence, etc.)
- A detailed introduction to prompt engineering and best practices, incremental prompting and anything else that could be revelatory and useful to scholars and creative writers
- A look at other GPT and AI-based tools that could be useful

Bruno brought up a number a points in his speech and I’d like to respond and elaborate/refute/confirm those:
{
Electronic - Generative - Interactive writing → ChatGPT

Issues:

- Lack of knowledge on how LLMs actually work
- Size vs Purpose
- Too much knowledge and power in the hands of few (companies) - once it’s out there, it’s out there = consent? Safety?
- lessons from Social Media: Fun, realization of harm to mental health and self, lack of regulation, use by children
- Plagiarism and Fraud - ethics
- Are humans obsolete? Will they be?

Human/Machine/Hybrid creation

******Important Questions and Considerations

- Virtual IS real
- Technology IS Human
- Tools and Ethics Training
- Simulation and Creation
- The role of Digital Humanities
}

I would also like to incorporate the paradox of Theseus’ ship in the hybridity of human+machine creation, a quick reference to Weisenbaum’s ELIZA when discussing how humans are vulnerable to human-like linguistic output and the uncanny valley effect, Rumi’s story of the elephant in the dark room, and also his lamentation by the reed. I have poems that I can provide you on these last two items. Any other similes , anecdotes or metaphors are also welcome.

I’d like to add the context that this piece is intended for the organization (https://etcl.uvic.ca/) that hosted Bueno [(“Navigating Digital Writing and AI: An Optimistic and Slightly Apocalyptic Examination”
Dr. Bernardo Bueno
The rapid advancement of artificial intelligence (AI) has brought about significant changes in the field of literature. Not long ago, the idea of an AI system writing acceptable poems, essays, and stories was considered distant. However, with the advent of ChatGPT and similar tools, we have seen a radical shift in this scenario. As writers, we can now engage with AI creatively and responsibly, but this development has also raised urgent ethical questions that demand investigation. This talk will explore the ethical implications of using AI in literature, including issues of copyright and the potential disappearance of human writers. We will also consider whether we will see AI-generated fiction being sold alongside human fiction, and whether hybrid (human + AI) fiction is a possibility. These questions are essential to understanding how creativity works and to determine if there should be limits to what digital technology can do in the literary chain of production and consumption.] I was present as a guest for the event . I would like to impress them enough for them to employ me as an academic researcher, and as such the voice of the essay should probably reflect that. As I wish to be able to steer the ETCL towards becoming an authority on the use of LLMs and particularly ChatGPT, I think a section on how the ETCL can contribute to the responsible, effective use and education on the subject would be instrumental.


Please read this chat between myself and ASSISTANT, and weave this entire conversation into a cohesive piece- don’t remove anything or summarize - I want everything treated in here particularly the unknowns, but reorganize the material from this point onward with the above in mind, and consolidate everything into a smooth, eloquent, easy-to-read yet in-depth treatment of unknowns in the use of LLMs, from technical details to creative implications.

{Bueno says that “we don’t know exactly how it works.” in reference to chatGPT. I have also heard the same being bandied about on neural networks. Is it true? What is it that we don’t know exactly?

## ASSISTANT
The statement “we don’t know exactly how it works” in reference to neural networks like ChatGPT is both true and not true, depending on the level of detail one is considering.

On one hand, we do understand the basic mechanics of how neural networks operate. They are composed of layers of interconnected nodes or “neurons,” and they “learn” patterns in data by adjusting the weights of these connections based on the error of their predictions. In the case of ChatGPT, a type of neural network called a transformer is used, which is particularly good at handling sequential data like text.

However, while we understand these basic mechanics, the complexity of these networks makes it difficult to understand why they make specific decisions. For example, when ChatGPT generates a piece of text, it’s challenging to pinpoint exactly why it chose a particular word or phrase. This is due to the high dimensionality of the network and the fact that the decision is influenced by the complex interplay of thousands or even millions of neuron weights.

Moreover, the training process of these models involves a large amount of randomness. The initial weights of the connections are usually set randomly, and the order in which training examples are presented to the model can also be random. This means that even with the same training data and network architecture, two models may end up behaving differently.

So, when people say “we don’t know exactly how it works,” they are often referring to this lack of interpretability and predictability. While we understand the overall process of how neural networks learn from data and make decisions, the specific reasons behind a particular decision made by a complex model like ChatGPT can be difficult to decipher. This is an active area of research in AI, with ongoing efforts to develop methods for making these models more interpretable and explainable.

## USER
what other unknowns exist along the way?

## ASSISTANT
In addition to the challenge of interpretability, there are several other “unknowns” or areas of uncertainty when it comes to neural networks and AI models like ChatGPT. Here are a few:

1. Bias and Fairness: AI models learn from data, and if the data they learn from is biased, the models can learn and perpetuate these biases. This can lead to unfair or discriminatory outcomes. However, detecting and mitigating bias in AI models is a complex task. It’s often difficult to know whether a model is biased, how severe the bias is, and how it’s affecting the model’s decisions.

2. Robustness and Generalization: While neural networks can perform exceptionally well on the types of data they were trained on, they can sometimes fail in unexpected ways when presented with slightly different data or situations. This is known as a lack of robustness or generalization. Understanding why this happens and how to improve the robustness of AI models is an ongoing research challenge.

3. Security and Privacy: As AI models become more powerful and are used in more applications, concerns about security and privacy are growing. For example, adversarial attacks can trick AI models into making incorrect decisions, and models like ChatGPT can inadvertently generate sensitive or private information if they were trained on such data. Addressing these concerns requires a deeper understanding of the vulnerabilities of AI models and the development of effective countermeasures.

4. Impact on Society: The widespread use of AI has far-reaching implications for society, many of which are not fully understood. For example, how will the automation of jobs by AI impact the economy and employment? How will the use of AI in decision-making affect fairness and equality? How will AI change the way we communicate, create, and think? These are complex, multifaceted questions that require interdisciplinary research to answer.

5. Ethics and Regulation: As AI becomes more integrated into our lives, ethical and regulatory questions become increasingly important. How should AI be regulated to ensure safety and accountability? How can we ensure that AI is used ethically and responsibly? These questions are still largely open, and finding the answers will require not only technical understanding, but also input from ethicists, policymakers, and society at large.

These are just a few of the “unknowns” in the field of AI. As research progresses, we will likely uncover new challenges and questions to explore.

## USER
can you think of any others?

## ASSISTANT
Certainly, there are several other areas of uncertainty and ongoing research in the field of AI and neural networks. Here are a few more:

1. Explainability: While we’ve touched on interpretability, it’s worth noting that explainability - the ability to provide clear, understandable reasons for a model’s decisions - is a distinct and equally important challenge. Even if we can interpret what a model is doing, it can be difficult to explain this in a way that non-experts can understand.

2. Causality: Most AI models, including neural networks, are fundamentally correlative, learning patterns in the data they’re trained on. However, correlation is not the same as causation, and models can struggle when applied to situations where understanding cause-and-effect relationships is important.

3. Long-term Dependability: AI models are often trained and evaluated on short-term tasks, but in many real-world applications, we need models that can operate reliably over long periods of time. Understanding how to ensure long-term dependability is a significant challenge.

4. Social Impact of AI-generated Content: As AI models become more capable of generating realistic content, concerns are growing about the potential for misuse. For example, AI-generated “deepfakes” can be used to create misleading or harmful content. Understanding the social impact of AI-generated content and how to mitigate potential harms is an important area of research.

5. AI and Mental Health: AI models like ChatGPT are increasingly being used in applications like mental health support. However, the impact of interacting with AI on individuals’ mental health is not fully understood.

6. Sustainability: Training large AI models requires significant computational resources, which can have a substantial environmental impact. Research is ongoing into how to make AI more sustainable.

7. AI in Education: AI has the potential to revolutionize education, but the implications of this are not fully understood. For example, how can we ensure that AI is used to support rather than replace teachers? How can we prevent the digital divide from widening as AI becomes more prevalent in education?

These are just a few of the many “unknowns” in the field of AI. As research progresses, we will likely uncover new challenges and questions to explore.

## USER
Now the million-dollar question: is it all bad that we don’t know? e.g. Isn’t that kind of the point of recreating neural structures with computers in the first place? Can we claim that we “know” how human neural networks fully work? And is there any aspect of this “unknowability” that we should be embracing?

## ASSISTANT
The “unknowns” in AI and neural networks, while they present challenges, are not inherently negative. In fact, they are a testament to the complexity and sophistication of these technologies, and they open up exciting avenues for research and discovery.

The endeavor to recreate neural structures with computers is indeed rooted in a desire to understand the workings of the human brain, one of the most complex and mysterious systems in existence. We are far from fully understanding how human neural networks work, and the same is true for their artificial counterparts. This parallel between human and artificial neural networks underscores the fact that “unknowability” is not unique to AI; it is a fundamental aspect of our quest to understand complex systems, whether they are biological or artificial.

Moreover, the “unknowability” of AI can be seen as a reflection of the richness and diversity of the patterns that these models can capture. The fact that we can’t fully predict or understand what a neural network will learn from a given dataset is, in a way, a testament to the model’s ability to learn from data in a flexible, adaptable manner, much like a human would.

This is not to say that we should blindly accept the “black box” nature of AI. Efforts to improve the interpretability and transparency of these models are crucial, particularly when they are used in high-stakes applications. However, we should also recognize that a certain degree of “unknowability” is inevitable when dealing with complex systems, and that this can be a source of creativity and discovery rather than a cause for concern.

In fact, embracing the “unknowns” of AI can lead to new insights and innovations. It can spur us to develop new methods for understanding these models, to explore novel applications, and to engage with important ethical and societal questions. It can also foster a sense of humility and caution, reminding us of the limits of our understanding and the need for responsible, thoughtful use of AI.

In the context of digital humanities, the “unknowns” of AI can be particularly fruitful. They can inspire new research questions, drive interdisciplinary collaboration, and enrich our understanding of digital culture. They can also inform the development of educational programs and resources, helping to equip individuals with the critical thinking skills they need to navigate an increasingly AI-augmented world.

So, while the “unknowns” of AI present challenges, they also offer opportunities. By embracing these “unknowns,” we can foster a more nuanced, critical, and creative engagement with AI.

## USER
And if unknowns and unknowability are to be embraced, what is the relation between creativity and the unknown?

## ASSISTANT
Creativity and the unknown share a profound and intricate relationship. At its core, creativity is the act of venturing into the unknown, of exploring uncharted territories and pushing the boundaries of what is known and familiar. It involves generating new ideas, concepts, or perspectives that deviate from established norms and conventions. In this sense, the unknown serves as both the canvas and the muse for creativity.

The unknown fuels creativity by providing a space for imagination and innovation. It invites us to question, to wonder, to dream, and to experiment. It challenges us to look beyond the status quo and to envision new possibilities. The unknown, therefore, is not just a void of uncertainty, but a realm of potential and promise.

Moreover, the act of engaging with the unknown can stimulate creative thinking. It requires us to be open-minded, flexible, and adaptable. It demands that we tolerate ambiguity, embrace uncertainty, and accept the risk of failure. These are all key traits of creative individuals and are essential for creative problem-solving and innovation.

The unknown also plays a crucial role in the creative process. Creativity often involves a phase of incubation, where ideas are allowed to simmer below the level of conscious awareness. This phase is shrouded in mystery, as it is not fully understood how this process leads to the emergence of creative insights. Yet, it is this dance with the unknown that often leads to the most surprising and innovative ideas.

In the context of AI, the unknown takes on additional layers of significance. AI technologies like ChatGPT can generate outputs that are unpredictable and surprising, even to their developers. This ability to venture into the unknown, to generate ideas that deviate from human expectations, can be a powerful tool for creativity. It can lead to novel and unexpected ideas, narratives, or perspectives that might not have been conceived by human minds alone.

However, this also underscores the importance of a critical and reflective approach to AI-augmented creativity. While AI can venture into the unknown, it does not possess an understanding or awareness of the implications of its outputs. Therefore, the role of the human in the creative process remains crucial, not only in guiding the AI, but also in interpreting, contextualizing, and evaluating its outputs.

In conclusion, the unknown is not just a challenge to be overcome, but a catalyst for creativity. By embracing the unknown, we can foster a more exploratory, imaginative, and critical engagement with AI and its role in creative endeavors.

## USER
Can you explain with academic eloquence, w
hat is generally not known about how LLMs work?

## ASSISTANT
Language models, particularly large language models (LLMs) like ChatGPT, are complex systems that operate based on intricate algorithms and vast amounts of data. While the fundamental principles and mechanisms underlying these models are well-understood, there are several aspects of their operation that remain less clear.

Firstly, the exact process by which LLMs generate specific outputs is not fully transparent. LLMs are trained on massive datasets, learning to predict the next word in a sentence based on patterns in the data. However, the precise sequence of computations and decisions that lead to a particular output is not readily interpretable. This is due to the high dimensionality and nonlinearity of the models, which makes it challenging to trace the contribution of individual inputs to the final output.

Secondly, LLMs exhibit emergent behaviors that are not explicitly programmed into the models. For instance, they can generate creative and coherent narratives, respond to prompts in contextually appropriate ways, and even exhibit a semblance of humor or wit. These behaviors emerge from the interaction of the model’s many layers and parameters, and are not directly attributable to specific aspects of the training data or the model architecture. Understanding how and why these behaviors emerge is a complex problem that is still an active area of research.

Thirdly, the relationship between the size of the model (i.e., the number of parameters) and its performance is not fully understood. While larger models generally perform better on a range of tasks, the reasons for this are not entirely clear. It is also unknown what the limits of this scaling trend might be, and whether there are diminishing returns or potential trade-offs associated with increasing model size.

Finally, LLMs can sometimes produce outputs that are unexpected or surprising, even to the developers of the models. They can generate plausible-sounding but incorrect or nonsensical answers, exhibit biases present in their training data, or respond to harmful or inappropriate prompts. Predicting and controlling these behaviors is a significant challenge, due to the complexity of the models and the diversity of potential inputs they can receive.

In conclusion, while we have a solid understanding of the basic principles of LLMs, there are many aspects of their operation that remain less clear. These unknowns present both challenges and opportunities for further research and exploration. They also underscore the importance of a critical and reflective approach to the use of these models, particularly in applications that have significant societal or ethical implications.

## USER
Thank you. What about the particulars of their operation is inscrutable? We already discussed the nebulous nature of how neural networks work; what else, from a structural or procedural perspective, can render LLMs into black boxes?

## ASSISTANT
The inscrutability of large language models (LLMs) like ChatGPT arises from several factors related to their structure, the nature of their training process, and the complexity of their operation.

1. High-Dimensional, Nonlinear Parameter Space: LLMs consist of millions, or even billions, of parameters. These parameters interact in complex, nonlinear ways to determine the model’s behavior. This high-dimensional parameter space is difficult to visualize or interpret, which contributes to the model’s inscrutability.

2. Opaque Training Process: LLMs are trained using a process called gradient descent, which iteratively adjusts the model’s parameters to minimize the difference between the model’s predictions and the actual data. This process is essentially a complex optimization problem in a high-dimensional space. The trajectory of this optimization process, and how it shapes the final model, is not fully understood.

3. Emergent Behavior: LLMs can exhibit behaviors that are not explicitly programmed into the model, but emerge from the interaction of its many layers and parameters. These emergent behaviors can be surprising and are not directly attributable to specific aspects of the training data or the model architecture.

4. Lack of Causal Reasoning: While LLMs can generate impressively human-like text, they do not understand or reason about the world in the way humans do. They do not have a model of the world, and they do not understand causality. This lack of understanding can lead to outputs that are plausible-sounding but incorrect or nonsensical.

5. Data Dependence: LLMs are entirely dependent on their training data. They learn to mimic the patterns in the data, but they do not understand the meaning of the text. This dependence on data can lead to biases and inaccuracies in the model’s output, and it also means that the model’s behavior can be influenced by the specific choice of training data.

6. Stochasticity: LLMs incorporate randomness in their operation, both during training and during text generation. This stochasticity can lead to variability in the model’s output, making it difficult to predict exactly what the model will generate given a particular input.

These factors contribute to the inscrutability of LLMs, making it challenging to fully understand and predict their behavior. This inscrutability is a significant challenge for the responsible use and governance of these models, and it underscores the importance of ongoing research and dialogue in this area.

## USER
Which part of this inscrutability can be resolved? Which part *should be resolved? Is there any part that wouldn’t necessarily benefit us? For instance, the complexity of the problem itself is the reason that we offload tasks to simulated neurons and neural structures, moving, as it were, from the “in vivo” wetware to the “in silico” hardware. Attempting to understand exactly why a certain output has arisen seems to be counter-intuitive.

## ASSISTANT
The inscrutability of large language models (LLMs) is a multifaceted issue, and different aspects of it present varying degrees of tractability and importance.

1. High-Dimensional, Nonlinear Parameter Space: The complexity of the parameter space in LLMs is a fundamental aspect of their design and is unlikely to be fully resolved. However, research into methods for visualizing and interpreting high-dimensional spaces could help us gain a better understanding of this aspect of LLMs.

2. Opaque Training Process: While the training process of LLMs is complex, there is ongoing research into understanding and improving this process. Techniques such as interpretability tools, model probing, and visualization can provide insights into how models learn and evolve during training.

3. Emergent Behavior: The emergence of unexpected behaviors in LLMs is a challenging issue. While it may not be possible to predict all emergent behaviors, developing robust testing and evaluation methods can help identify and mitigate potentially harmful behaviors.

4. Lack of Causal Reasoning: This is a fundamental limitation of current LLMs and is a significant area of ongoing research. Incorporating causal reasoning into LLMs could greatly improve their ability to generate accurate and meaningful outputs.

5. Data Dependence: The dependence of LLMs on their training data is both a strength and a weakness. While it allows them to learn from a wide range of data, it also makes them susceptible to biases in the data. Efforts to curate diverse and unbiased training datasets, as well as research into methods for mitigating bias in model outputs, are crucial in this regard.

6. Stochasticity: The randomness in LLMs’ operation is a fundamental aspect of their design and is unlikely to be eliminated. However, understanding the role of stochasticity in model behavior can help in developing methods for controlling and managing it.

While some aspects of LLMs’ inscrutability may be inherent to their design and operation, others can and should be addressed. The goal should not necessarily be to eliminate inscrutability, but to manage it in a way that ensures the responsible and beneficial use of these models. This involves a combination of technical solutions, such as improved interpretability and transparency, and non-technical measures, such as robust governance mechanisms and ethical guidelines.

Moreover, it’s important to recognize that some degree of inscrutability is not only inevitable but also potentially beneficial. The ability of LLMs to solve complex problems and generate creative outputs often stems from their ability to navigate high-dimensional spaces and identify patterns that may not be apparent to humans. This “unknowability” can be a source of innovation and discovery, as long as it is managed responsibly.

## USER
That is excellent. Can we expand these two last paragraphs within the context of DH and ETC and how they can help?

## ASSISTANT
Certainly. The inscrutability of large language models (LLMs) presents a unique challenge and opportunity for digital humanities (DH) and research organizations like the Electronic Textual Cultures Lab (ETCL).

While some aspects of LLMs’ inscrutability may be inherent to their design and operation, others can and should be addressed. The goal should not necessarily be to eliminate inscrutability, but to manage it in a way that ensures the responsible and beneficial use of these models. This involves a combination of technical solutions, such as improved interpretability and transparency, and non-technical measures, such as robust governance mechanisms and ethical guidelines.

DH and organizations like the ETCL can play a pivotal role in this endeavor. With their interdisciplinary expertise and critical perspectives, they can contribute to the development of interpretability tools and methodologies that enhance our understanding of LLMs. They can also lead efforts to establish ethical guidelines and governance mechanisms for the use of LLMs in research and creative practice.

Moreover, it’s important to recognize that some degree of inscrutability is not only inevitable but also potentially beneficial. The ability of LLMs to solve complex problems and generate creative outputs often stems from their ability to navigate high-dimensional spaces and identify patterns that may not be apparent to humans. This “unknowability” can be a source of innovation and discovery, as long as it is managed responsibly.

Here again, DH and organizations like the ETCL can play a crucial role. They can help us navigate the balance between leveraging the creative potential of LLMs and ensuring their responsible use. They can foster a critical understanding of LLMs, promoting an approach that embraces their creative potential while being mindful of their limitations and ethical implications.

In this way, DH and organizations like the ETCL can contribute to the development of a responsible and innovative approach to using LLMs in writing and creativity. They can help us harness the power of these models while navigating the challenges and opportunities they present.

## USER
What specific work or research can they do? Could you expand the above and exemplify please?

## ASSISTANT
Here are two examples of projects in the digital humanities that involve large language models:

1. Model of Models: This is a platform for collaborative language research and teaching, bringing together humanistic methods with technical innovations. The project is run by the UC Cincinnati Digital Scholarship Center (DSC) and funded by the Andrew W. Mellon Foundation. The platform allows users to create their own models and provides a tutorial for beginners. One of their featured projects is “CO2 Emissions and Race,” which includes maps and visualizations showing the relationships between race, income, and CO2 emissions for select cities.

2. High Performance Language Technologies (HPLT): This project combines petabytes of natural language data with large-scale model training. The HPLT project aims to create efficient and high-quality language and translation models using high-performance computing. The project emphasizes FAIR (Findable, Accessible, Interoperable, and Reusable) principles, transparency, high quality, and efficiency in its models. The consortium partners include several universities and research institutions.

These projects demonstrate how digital humanities research organizations can contribute to the understanding and responsible use of AI in academic research and writing. They show how these organizations can conduct research on AI and textuality, develop best practices for using AI in humanities research, and facilitate dialogues on the ethical and societal implications of AI.

For instance, digital humanities research organizations could conduct studies to understand the inscrutability of AI models, exploring questions such as why certain outputs are generated and how different inputs affect the outputs. They could also investigate the ethical implications of this inscrutability, such as the potential for bias or misinformation.

Moreover, these organizations could develop guidelines and best practices for using AI models in humanities research. This could include recommendations for how to use AI responsibly, how to interpret the outputs of AI models, and how to address potential ethical issues.

Furthermore, these organizations could facilitate dialogues and discussions about the role of AI in humanities research. This could involve hosting workshops, seminars, or conferences where researchers, practitioners, and policymakers can share their insights and experiences, discuss challenges and opportunities, and collaborate on potential solutions.

By undertaking these initiatives, digital humanities research organizations can play a pivotal role in navigating the future of AI, creativity, and writing. They can help ensure that AI is used responsibly and effectively in academic research and writing, and they can contribute to the broader understanding and discussion of AI’s role in our society.
}
## USER
Thank you. From a writing perspective, and as part of the work we’re doing, could you consolidate what we have discussed from the “aside” I mentioned up till now? I want to incorporate it into a new section in the introduction entitled “We don’t know what we don’t know…”